00000100:00020000:3.0F:1496184918.855488:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184918.866587:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:4.0F:1496184918.888565:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104076800 msgsize 520 00000100:00100000:4.0:1496184918.888575:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184918.888576:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184918.891036:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184918.902136:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:4.0:1496184918.924959:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184918.924966:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184918.924967:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184918.926681:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184918.937760:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00000040:4.0:1496184918.959619:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb450 x1568861104076864 msgsize 520 00000100:00100000:4.0:1496184918.959630:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184918.959632:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184918.962428:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184918.973451:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00100000:4.0:1496184918.995576:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104076896 00000020:00000040:4.0:1496184918.995580:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184918.995582:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184918.995601:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42650 00000100:00000040:4.0:1496184918.995603:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184918.995606:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc050 x1568861104076896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184938 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184918.995613:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076896:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184918.995621:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184918.995629:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184918.995632:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184918.995636:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184918 last 1496184917 00000001:00000040:4.0:1496184918.995642:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184918.995644:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184918.995647:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184918.995654:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184918.995658:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184918.995659:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184918.995660:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184918.995663:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184918.995666:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184918.995671:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184918.995673:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184918.995676:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184918.995678:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184918.995682:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184918.995685:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184918.995687:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184918.995689:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184918.995694:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104076896 00000100:00000040:4.0:1496184918.995700:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184918.995703:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184918.995718:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184918.995721:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc050 x1568861104076896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184938 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184918.995727:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076896:12345-10.37.248.124@o2ib1:8 Request procesed in 114us (203us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184918.995731:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42650 00000100:00000040:4.0:1496184918.995733:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184918.995736:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184918.996285:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184918.996288:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184918.996290:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184918.996295:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184918.996305:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/0 e 0 to 0 dl 1496184961 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184918.996312:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184918.996316:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:4.0:1496184918.996327:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56755 00000100:00000040:4.0:1496184918.996328:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184918.996331:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/0 e 0 to 0 dl 1496184961 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184918.996337:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184918.996389:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184918.998093:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba600 00000100:00020000:3.0:1496184919.009129:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba600 00010000:00020000:4.0:1496184919.020157:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/448 e 0 to 0 dl 1496184961 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184919.020205:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184919.020211:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184919.020214:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/448 e 0 to 0 dl 1496184961 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184919.020219:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184919.020221:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/448 e 0 to 0 dl 1496184961 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184919.020227:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23890us (23988us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184919.020232:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56755 00000100:00000040:4.0:1496184919.020234:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184919.020237:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:706/0 lens 608/448 e 0 to 0 dl 1496184961 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184919.020243:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.020245:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:4.0:1496184919.020249:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184919.030555:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104076928 00000020:00000040:4.0:1496184919.030560:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184919.030562:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184919.030576:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42651 00000100:00000040:4.0:1496184919.030578:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.030581:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc450 x1568861104076928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184919.030588:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076928:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184919.030594:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184919.030599:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184919.030602:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184919.030604:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000001:00000040:4.0:1496184919.030608:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184919.030610:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184919.030611:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184919.030616:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184919.030619:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184919.030620:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184919.030621:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184919.030622:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.030624:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.030628:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184919.030629:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184919.030631:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.030633:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184919.030636:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184919.030638:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184919.030641:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184919.030642:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184919.030646:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104076928 00000100:00000040:4.0:1496184919.030650:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.030652:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.030664:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.030667:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc450 x1568861104076928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184919.030673:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076928:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (164us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184919.030677:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42651 00000100:00000040:4.0:1496184919.030679:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184919.030682:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184919.031224:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184919.031228:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184919.031229:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184919.031234:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184919.031242:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184919.031249:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.031252:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:4.0:1496184919.031261:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56756 00000100:00000040:4.0:1496184919.031263:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.031265:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184919.031272:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184919.031315:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184919.032806:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba800 00000100:00020000:3.0:1496184919.043855:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba800 00000100:00000040:2.0F:1496184919.054939:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2450 x1568861104076960 msgsize 520 00000100:00100000:2.0:1496184919.054949:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184919.054951:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00010000:00020000:4.0:1496184919.054961:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00100000:3.0:1496184919.054969:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104076960 00000020:00000040:3.0:1496184919.054972:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184919.054975:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184919.054997:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5944 00000100:00000040:3.0:1496184919.055000:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184919.055004:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2450 x1568861104076960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184919.055012:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104076960:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184919.055015:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184919.055021:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184919.055023:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00010000:00000080:4.0:1496184919.055024:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184919.055029:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184919.055032:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184919.055033:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:4.0:1496184919.055038:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+10:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23766us (23860us total) trans 0 rc -110/-999 00010000:02000400:3.0:1496184919.055038:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184919.055041:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000100:00100000:4.0:1496184919.055044:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56756 00000100:00000040:4.0:1496184919.055046:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000001:00000040:3.0:1496184919.055049:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00002000:00100000:4.0:1496184919.055050:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000001:00000040:3.0:1496184919.055051:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184919.055054:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184919.055056:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.055059:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00002000:00100000:3.0:1496184919.055061:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000020:00000040:4.0:1496184919.055063:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184919.055066:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.055070:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.055071:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184919.055072:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.055075:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.055080:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.055082:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184919.055085:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.055087:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.055091:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184919.055094:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184919.055097:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.055098:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184919.055105:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104076960 00000100:00000040:3.0:1496184919.055112:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.055115:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.055133:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.055135:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2450 x1568861104076960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184919.055141:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076960:12345-10.37.248.124@o2ib1:8 Request procesed in 130us (201us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184919.055146:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5944 00000100:00000040:3.0:1496184919.055148:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.055158:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184919.056172:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184919.056175:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184919.056177:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184919.056184:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184919.056192:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184919.056199:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.056201:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:4.0:1496184919.056211:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56757 00000100:00000040:4.0:1496184919.056213:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.056217:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184919.056224:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184919.056268:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184919.061260:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba800 00000100:00020000:3.0:1496184919.072273:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba800 00010000:00020000:4.0:1496184919.083297:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184919.083347:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184919.083352:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184919.083355:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184919.083360:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184919.083363:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184919.083369:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 27145us (27241us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184919.083374:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56757 00000100:00000040:4.0:1496184919.083375:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184919.083379:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/448 e 0 to 0 dl 1496184962 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184919.083385:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.083388:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:4.0:1496184919.083391:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184919.094053:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104076992 00000020:00000040:4.0:1496184919.094056:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184919.094058:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184919.094073:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42652 00000100:00000040:4.0:1496184919.094075:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.094080:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc850 x1568861104076992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184919.094088:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076992:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184919.094095:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184919.094102:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184919.094105:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184919.094108:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000001:00000040:4.0:1496184919.094113:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184919.094115:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184919.094117:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184919.094122:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184919.094126:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184919.094127:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184919.094128:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184919.094129:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.094131:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.094136:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184919.094138:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184919.094141:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.094143:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184919.094147:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184919.094149:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184919.094151:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184919.094153:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184919.094157:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104076992 00000100:00000040:4.0:1496184919.094162:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.094165:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.094178:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184919.094181:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc850 x1568861104076992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184919.094186:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104076992:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (181us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184919.094191:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42652 00000100:00000040:4.0:1496184919.094192:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184919.094195:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184919.094785:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184919.094788:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184919.094790:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184919.094796:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184919.094804:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184919.094811:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184919.094814:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000100:00100000:4.0:1496184919.094823:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56758 00000100:00000040:4.0:1496184919.094825:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184919.094827:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:707/0 lens 608/0 e 0 to 0 dl 1496184962 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184919.094834:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184919.094879:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184919.096532:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b8000 00000100:00020000:3.0:1496184919.096557:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b8000 00000100:00020000:3.0:1496184919.127548:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184919.127571:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184919.157780:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184919.157806:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184919.187638:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184919.187655:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184919.217629:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.228755:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00000040:0.0F:1496184919.250134:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4050 x1568861104077152 msgsize 520 00000100:00100000:0.0:1496184919.250145:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184919.250147:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184919.250195:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104077152 00000020:00000040:3.0:1496184919.250199:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184919.250202:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184919.250230:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5945 00000100:00000040:3.0:1496184919.250232:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.250237:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4050 x1568861104077152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184919.250246:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104077152:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184919.250257:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184919.250268:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184919.250273:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184919.250276:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000001:00000040:3.0:1496184919.250284:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184919.250287:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184919.250289:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184919.250298:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184919.250302:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.250304:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.250305:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184919.250306:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.250309:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.250317:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.250319:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184919.250322:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.250325:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.250329:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184919.250332:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184919.250335:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.250336:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184919.250344:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104077152 00000100:00000040:3.0:1496184919.250351:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.250354:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.250373:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.250376:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4050 x1568861104077152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184919.250382:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104077152:12345-10.37.248.124@o2ib1:8 Request procesed in 137us (247us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184919.250387:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5945 00000100:00000040:3.0:1496184919.250389:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.250395:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184919.252623:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.263682:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.287923:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.298990:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.323583:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184919.334617:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00000100:00000040:2.0:1496184919.340920:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184919.340931:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184919.340951:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.340963:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76a400] to pc [ptlrpcd_00_01:1] req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.340971:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184919.340975:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184919.340978:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.340984:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76b600] to pc [ptlrpcd_00_02:2] req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.340991:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184919.340993:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184919.340997:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.341002:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76b300] to pc [ptlrpcd_00_03:3] req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:1.0F:1496184919.341007:0:13689:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00000040:2.0:1496184919.341008:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 5000 (4297358369) 00000100:00000040:0.0:1496184919.341011:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184919.341019:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184919.341024:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:13691:1568861159710016:10.37.248.197@o2ib1:400 00000100:00100000:1.0:1496184919.341028:0:13689:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:d86c02e9-1939-02b8-4331-d3612fcf0106:13689:1568861159710000:10.37.248.196@o2ib1:400 00000100:00000040:1.0:1496184919.341044:0:13689:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184919.341044:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184919.341051:0:13689:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:0.0:1496184919.341051:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00100000:0.0:1496184919.341078:0:13691:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00000040:0.0:1496184919.341082:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184919.341088:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710032:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184919.341093:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184919.341098:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184919.345658:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.345671:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184919.345680:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.345695:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184919.345716:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184919.345727:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184919.345750:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184919.345757:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:1.0:1496184919.345762:0:13689:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:d86c02e9-1939-02b8-4331-d3612fcf0106:13689:1568861159710000:10.37.248.196@o2ib1:400 00000100:00000040:1.0:1496184919.345765:0:13689:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76a400 x1568861159710000/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:1.0:1496184919.345773:0:13689:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000100:00000040:0.0:1496184919.345800:0:13691:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:0.0:1496184919.345805:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:0.0:1496184919.345812:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:0.0:1496184919.345818:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710032:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184919.345822:0:13691:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000040:0.0:1496184919.345824:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:0.0:1496184919.345829:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:0.0:1496184919.345834:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:13691:1568861159710016:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184919.345837:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76b300 x1568861159710032/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:0.0:1496184919.345844:0:13691:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:0.0:1496184919.345848:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76b600 x1568861159710016/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184926 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:0.0:1496184919.345853:0:13691:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000100:00020000:3.0:1496184919.359192:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184919.370270:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184919.394971:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184919.406009:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184919.430203:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184919.441250:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00000100:00000040:1.0:1496184919.452355:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2c50 x1568861104077344 msgsize 520 00000100:00100000:1.0:1496184919.452364:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184919.452366:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184919.452385:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104077344 00000020:00000040:3.0:1496184919.452389:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184919.452393:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184919.452417:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5946 00000100:00000040:3.0:1496184919.452420:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.452425:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2c50 x1568861104077344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184919.452434:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104077344:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184919.452447:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184919.452459:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184919.452464:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184919.452467:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000001:00000040:3.0:1496184919.452483:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184919.452486:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184919.452489:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184919.452497:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184919.452502:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.452504:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.452505:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184919.452506:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.452509:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.452515:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.452518:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184919.452521:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.452523:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.452531:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184919.452534:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184919.452536:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.452538:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184919.452546:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104077344 00000100:00000040:3.0:1496184919.452554:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.452558:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.452577:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.452580:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2c50 x1568861104077344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184919.452586:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104077344:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184919.452592:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5946 00000100:00000040:3.0:1496184919.452594:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.452599:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184919.461350:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184919.472349:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184919.496537:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184919.496561:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184919.528091:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184919.528116:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184919.558990:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184919.559001:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184919.589094:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184919.589110:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184919.619025:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184919.630186:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184919.654218:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184919.665279:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184919.689586:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184919.700698:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184919.725010:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184919.736056:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184919.760386:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184919.771442:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184919.795650:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184919.806704:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184919.830832:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184919.841871:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00000100:00000040:2.0:1496184919.852983:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3050 x1568861104077744 msgsize 520 00000100:00100000:2.0:1496184919.852991:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184919.852993:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184919.853013:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104077744 00000020:00000040:3.0:1496184919.853017:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184919.853020:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184919.853045:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5947 00000100:00000040:3.0:1496184919.853048:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.853053:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3050 x1568861104077744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184939 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184919.853062:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104077744:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184919.853075:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184919.853088:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184919.853092:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184919.853096:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184919 last 1496184919 00000001:00000040:3.0:1496184919.853112:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184919.853116:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184919.853118:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184919.853126:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184919.853131:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.853133:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.853137:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184919.853138:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184919.853141:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.853147:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.853149:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184919.853152:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.853155:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184919.853160:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184919.853163:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184919.853165:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184919.853167:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184919.853175:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104077744 00000100:00000040:3.0:1496184919.853183:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.853186:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.853206:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184919.853208:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3050 x1568861104077744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184939 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184919.853214:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104077744:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184919.853220:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5947 00000100:00000040:3.0:1496184919.853222:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184919.853228:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184919.861523:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184919.872520:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184919.897009:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184919.897032:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00020000:0.0:1496184919.928882:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:0.0:1496184919.928904:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00000100:00020000:1.0:1496184919.959323:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:1.0:1496184919.959348:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184919.989663:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184919.989679:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184920.019954:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184920.031095:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184920.055644:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184920.066709:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184920.091090:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184920.102177:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184920.126744:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184920.137807:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184920.162278:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184920.173364:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184920.197805:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184920.208836:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184920.232946:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184920.243987:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00000100:00000040:1.0:1496184920.255094:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2050 x1568861104078128 msgsize 520 00000100:00100000:1.0:1496184920.255103:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184920.255104:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184920.255124:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078128 00000020:00000040:3.0:1496184920.255128:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184920.255132:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184920.255156:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5948 00000100:00000040:3.0:1496184920.255158:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184920.255164:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2050 x1568861104078128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184920.255173:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104078128:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184920.255186:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184920.255199:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184920.255203:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184920.255207:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:3.0:1496184920.255215:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184920.255218:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184920.255220:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184920.255240:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184920.255244:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.255246:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184920.255247:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184920.255248:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184920.255252:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.255257:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.255259:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184920.255263:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.255265:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.255271:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184920.255274:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184920.255276:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184920.255278:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184920.255286:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078128 00000100:00000040:3.0:1496184920.255293:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.255297:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.255316:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.255319:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2050 x1568861104078128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184920.255325:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078128:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184920.255330:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5948 00000100:00000040:3.0:1496184920.255332:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.255338:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184920.261418:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184920.272426:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184920.296624:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184920.296649:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184920.327084:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184920.327108:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184920.357262:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184920.357273:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184920.387616:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184920.387643:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184920.417831:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184920.428999:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184920.453265:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184920.464325:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184920.488730:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184920.499808:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184920.524799:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184920.535864:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184920.560346:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184920.571414:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184920.595193:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184920.606259:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184920.630627:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184920.641714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00000100:00000040:1.0:1496184920.652828:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1850 x1568861104078512 msgsize 520 00000100:00100000:1.0:1496184920.652839:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184920.652841:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184920.652860:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078512 00000020:00000040:3.0:1496184920.652864:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184920.652868:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184920.652911:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5949 00000100:00000040:3.0:1496184920.652913:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184920.652918:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1850 x1568861104078512/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184920.652927:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104078512:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184920.652939:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184920.652953:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184920.652958:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184920.652962:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:3.0:1496184920.652970:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184920.652973:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184920.652975:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184920.652984:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184920.652988:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.652990:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000040:3.0:1496184920.652992:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184920.652993:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184920.652996:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.653002:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.653004:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184920.653007:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.653009:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.653014:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184920.653017:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184920.653020:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184920.653022:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184920.653030:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078512 00000100:00000040:3.0:1496184920.653037:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.653041:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.653060:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.653062:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1850 x1568861104078512/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184920.653069:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078512:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184920.653075:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5949 00000100:00000040:3.0:1496184920.653077:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.653083:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184920.654009:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184920.654012:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184920.654015:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184920.654020:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184920.654028:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184920.654041:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.654043:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000100:00100000:4.0:1496184920.654051:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56805 00000100:00000040:4.0:1496184920.654053:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.654056:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184920.654062:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184920.654109:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184920.661255:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9e00 00000100:00020000:3.0:1496184920.672345:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9e00 00010000:00020000:4.0:1496184920.683737:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184920.683783:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184920.683788:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184920.683791:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184920.683796:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184920.683799:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184920.683805:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 29742us (29809us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184920.683809:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56805 00000100:00000040:4.0:1496184920.683811:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184920.683815:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184920.683820:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.683825:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000020:00000040:4.0:1496184920.683829:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184920.692964:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078544 00000020:00000040:4.0:1496184920.692966:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184920.692968:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184920.692980:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42695 00000100:00000040:4.0:1496184920.692982:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.692985:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1850 x1568861104078544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184920.692992:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078544:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184920.692998:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184920.693004:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184920.693015:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184920.693018:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:4.0:1496184920.693022:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184920.693024:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184920.693025:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184920.693030:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184920.693033:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184920.693034:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184920.693035:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184920.693036:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.693038:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.693042:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184920.693044:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184920.693046:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.693048:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184920.693053:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184920.693055:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184920.693057:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184920.693058:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184920.693062:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078544 00000100:00000040:4.0:1496184920.693066:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.693076:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.693088:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.693090:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1850 x1568861104078544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184920.693096:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078544:12345-10.37.248.124@o2ib1:8 Request procesed in 104us (158us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184920.693101:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42695 00000100:00000040:4.0:1496184920.693102:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184920.693105:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184920.693554:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184920.693555:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184920.693556:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184920.693559:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184920.693565:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184920.693571:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.693573:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000100:00100000:4.0:1496184920.693581:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56806 00000100:00000040:4.0:1496184920.693582:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.693584:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184920.693590:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184920.693633:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184920.695176:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9200 00000100:00020000:3.0:1496184920.706283:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9200 00010000:00020000:4.0:1496184920.717379:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184920.717424:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184920.717429:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184920.717432:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184920.717437:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184920.717440:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184920.717446:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23855us (23907us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184920.717450:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56806 00000100:00000040:4.0:1496184920.717452:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184920.717456:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184920.717461:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.717464:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000020:00000040:4.0:1496184920.717467:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184920.726986:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078576 00000020:00000040:4.0:1496184920.726989:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184920.726990:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184920.727005:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42696 00000100:00000040:4.0:1496184920.727006:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.727009:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0450 x1568861104078576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184920.727016:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078576:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184920.727024:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184920.727029:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184920.727032:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184920.727035:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:4.0:1496184920.727039:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184920.727040:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184920.727050:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184920.727055:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184920.727057:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184920.727059:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184920.727060:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184920.727060:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.727063:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.727067:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184920.727068:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184920.727070:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.727072:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184920.727075:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184920.727077:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184920.727079:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184920.727080:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184920.727084:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078576 00000100:00000040:4.0:1496184920.727088:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.727090:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.727102:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184920.727113:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0450 x1568861104078576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184920.727120:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078576:12345-10.37.248.124@o2ib1:8 Request procesed in 104us (179us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184920.727124:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42696 00000100:00000040:4.0:1496184920.727126:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184920.727129:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184920.727869:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184920.727872:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184920.727874:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184920.727880:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184920.727888:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184920.727903:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.727906:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000100:00100000:4.0:1496184920.727916:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56807 00000100:00000040:4.0:1496184920.727917:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184920.727920:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/0 e 0 to 0 dl 1496184963 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184920.727926:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184920.727969:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184920.729414:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b8c00 00000100:00020000:3.0:1496184920.740517:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b8c00 00010000:00020000:4.0:1496184920.751678:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184920.751723:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184920.751728:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184920.751731:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184920.751736:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184920.751741:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184920.751747:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23820us (23922us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184920.751751:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56807 00000100:00000040:4.0:1496184920.751753:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184920.751757:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:708/0 lens 608/448 e 0 to 0 dl 1496184963 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184920.751762:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184920.751765:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000020:00000040:4.0:1496184920.751768:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:0.0:1496184920.763661:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4400 00000100:00020000:0.0:1496184920.774714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4400 00000100:00000040:1.0:1496184920.785902:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2850 x1568861104078640 msgsize 520 00000100:00100000:1.0:1496184920.785911:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184920.785913:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184920.785934:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078640 00000020:00000040:2.0:1496184920.785938:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184920.785942:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184920.785967:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5950 00000100:00000040:2.0:1496184920.785970:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184920.785975:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2850 x1568861104078640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184920.785983:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078640:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184920.785996:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184920.786007:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184920.786012:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184920.786016:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:2.0:1496184920.786024:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184920.786027:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184920.786029:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184920.786037:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184920.786041:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184920.786044:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184920.786045:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184920.786046:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184920.786049:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184920.786054:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184920.786057:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184920.786062:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184920.786064:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184920.786069:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184920.786072:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184920.786074:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184920.786076:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184920.786085:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078640 00000100:00000040:2.0:1496184920.786092:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184920.786096:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184920.786125:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184920.786127:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2850 x1568861104078640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184920.786134:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078640:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (231us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184920.786139:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5950 00000100:00000040:2.0:1496184920.786141:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184920.786146:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184920.788523:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4400 00000100:00020000:3.0:1496184920.799536:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4400 00000100:00000040:4.0:1496184920.821225:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184920.821236:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.821245:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184920.822928:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4400 00000100:00020000:3.0:1496184920.833973:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4400 00000100:00000040:1.0:1496184920.845069:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1450 x1568861104078704 msgsize 520 00000100:00100000:1.0:1496184920.845086:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184920.845088:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184920.845107:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078704 00000020:00000040:3.0:1496184920.845111:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184920.845115:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184920.845139:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5951 00000100:00000040:3.0:1496184920.845141:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184920.845146:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1450 x1568861104078704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184940 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184920.845156:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104078704:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184920.845167:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184920.845179:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184920.845184:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184920.845187:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184920 last 1496184919 00000001:00000040:3.0:1496184920.845195:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184920.845198:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184920.845203:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184920.845211:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184920.845215:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.845217:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184920.845219:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184920.845220:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184920.845229:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.845235:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.845237:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184920.845241:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.845243:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184920.845248:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184920.845251:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184920.845254:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184920.845255:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184920.845264:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078704 00000100:00000040:3.0:1496184920.845272:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.845275:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.845294:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184920.845297:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1450 x1568861104078704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184940 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184920.845303:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078704:12345-10.37.248.124@o2ib1:8 Request procesed in 148us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184920.845309:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5951 00000100:00000040:3.0:1496184920.845311:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184920.845316:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184920.848056:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4400 00000100:00020000:3.0:1496184920.859069:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4400 00000100:00000040:4.0:1496184920.884446:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1050 x1568861104078736 msgsize 520 00000100:00100000:4.0:1496184920.884453:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.884454:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184920.885106:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ac50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184920.885113:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.885115:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184920.886615:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184920.897651:0:6132:0:(events.c:347:request_in_callback()) incoming req@ffff8807f9348450 x1568861152397360 msgsize 224 00000100:00020000:3.0:1496184920.897658:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00100000:1.0:1496184920.897659:0:6132:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:1.0:1496184920.897660:0:6132:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184920.897711:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397360 00000020:00000040:2.0:1496184920.897715:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:2.0:1496184920.897717:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:2.0:1496184920.897726:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:2.0:1496184920.897739:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 948 00000100:00000040:2.0:1496184920.897741:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:2.0:1496184920.897745:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f9348450 x1568861152397360/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:671/0 lens 224/0 e 0 to 0 dl 1496184926 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184920.897753:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397360:12345-10.37.248.197@o2ib1:13 00002000:00000024:2.0:1496184920.897770:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:2.0:1496184920.897776:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397360 00000100:00000040:2.0:1496184920.897784:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:2.0:1496184920.897788:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:2.0:1496184920.897804:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:2.0:1496184920.897806:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f9348450 x1568861152397360/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:671/0 lens 224/336 e 0 to 0 dl 1496184926 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:2.0:1496184920.897812:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397360:12345-10.37.248.197@o2ib1:13 Request procesed in 60us (160us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184920.897817:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 948 00000100:00000040:2.0:1496184920.897819:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:2.0:1496184920.897823:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00000100:00020000:3.0:1496184920.922359:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184920.933396:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00000040:4.0:1496184920.955196:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd8c50 x1568861104078800 msgsize 520 00000100:00100000:4.0:1496184920.955203:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.955204:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184920.955937:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184920.955944:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.955945:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184920.957970:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184920.969026:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00000040:4.0:1496184920.990948:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104078832 msgsize 520 00000100:00100000:4.0:1496184920.990955:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.990958:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184920.991717:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184920.991724:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184920.991725:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184920.993601:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184921.004642:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:4.0:1496184921.026039:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104078864 msgsize 520 00000100:00100000:4.0:1496184921.026046:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.026047:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184921.026758:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.026765:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.026766:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.028925:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184921.039985:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:2.0:1496184921.051079:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1c50 x1568861104078896 msgsize 520 00000100:00100000:2.0:1496184921.051088:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184921.051090:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184921.051112:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104078896 00000020:00000040:3.0:1496184921.051116:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184921.051128:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184921.051154:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5952 00000100:00000040:3.0:1496184921.051156:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184921.051161:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1c50 x1568861104078896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184941 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184921.051170:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104078896:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184921.051182:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184921.051194:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184921.051199:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184921.051203:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184921 last 1496184921 00000001:00000040:3.0:1496184921.051211:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184921.051214:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184921.051216:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184921.051224:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184921.051229:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.051233:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.051235:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184921.051236:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.051239:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.051244:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.051247:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184921.051250:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.051252:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.051258:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184921.051261:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184921.051263:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.051264:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184921.051280:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104078896 00000100:00000040:3.0:1496184921.051288:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.051292:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.051312:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.051314:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1c50 x1568861104078896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184941 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184921.051320:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104078896:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184921.051326:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5952 00000100:00000040:3.0:1496184921.051340:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.051345:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184921.052261:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.052269:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.052270:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.061579:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184921.072580:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:4.0:1496184921.089463:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9050 x1568861104078928 msgsize 520 00000100:00100000:4.0:1496184921.089470:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.089471:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.092252:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184921.103324:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184921.127246:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184921.138294:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:4.0:1496184921.160037:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9850 x1568861104078992 msgsize 520 00000100:00100000:4.0:1496184921.160044:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.160045:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184921.160795:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600cc50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.160809:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.160811:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.162660:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184921.173697:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184921.193371:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184921.204401:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:4.0:1496184921.225840:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bda450 x1568861104079056 msgsize 520 00000100:00100000:4.0:1496184921.225847:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.225848:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184921.226535:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.226542:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.226543:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.228138:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184921.239177:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:2.0:1496184921.250231:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1050 x1568861104079088 msgsize 520 00000100:00100000:2.0:1496184921.250240:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184921.250242:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184921.250271:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079088 00000020:00000040:3.0:1496184921.250275:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184921.250278:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184921.250303:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5953 00000100:00000040:3.0:1496184921.250305:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184921.250310:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1050 x1568861104079088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184941 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184921.250319:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104079088:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184921.250330:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184921.250343:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184921.250348:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184921.250352:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184921 last 1496184921 00000001:00000040:3.0:1496184921.250360:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184921.250363:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184921.250366:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184921.250373:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184921.250378:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.250380:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.250381:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184921.250382:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.250385:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.250393:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.250396:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184921.250399:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.250401:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.250406:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184921.250416:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184921.250419:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.250421:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184921.250429:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079088 00000100:00000040:3.0:1496184921.250436:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.250440:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.250459:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.250462:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1050 x1568861104079088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184941 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184921.250468:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079088:12345-10.37.248.124@o2ib1:8 Request procesed in 150us (236us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184921.250474:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5953 00000100:00000040:3.0:1496184921.250476:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.250481:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184921.251430:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.251437:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.251440:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.261390:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184921.272398:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:4.0:1496184921.289399:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdac50 x1568861104079120 msgsize 520 00000100:00100000:4.0:1496184921.289406:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.289407:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184921.290106:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.290113:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.290114:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.291806:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184921.302849:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:2.0:1496184921.323195:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0450 x1568861104079152 msgsize 520 00000100:00100000:2.0:1496184921.323204:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184921.323206:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184921.323226:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079152 00000020:00000040:3.0:1496184921.323238:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184921.323242:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184921.323269:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5954 00000100:00000040:3.0:1496184921.323272:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.323277:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0450 x1568861104079152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184941 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184921.323288:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079152:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184921.323300:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184921.323313:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184921.323318:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184921.323321:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184921 last 1496184921 00000001:00000040:3.0:1496184921.323329:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184921.323333:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184921.323335:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184921.323344:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184921.323348:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.323350:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.323351:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184921.323352:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.323355:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.323360:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.323363:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184921.323366:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.323368:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.323374:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184921.323377:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184921.323379:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.323381:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184921.323389:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079152 00000100:00000040:3.0:1496184921.323397:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.323401:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.323421:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.323423:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0450 x1568861104079152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184941 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184921.323430:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079152:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (234us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184921.323435:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5954 00000100:00000040:3.0:1496184921.323437:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.323442:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184921.324004:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600dc50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.324011:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.324013:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.326112:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184921.337136:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184921.361862:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184921.372912:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:4.0:1496184921.394775:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104079216 msgsize 520 00000100:00100000:4.0:1496184921.394782:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.394783:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184921.395491:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ec50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184921.395498:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184921.395499:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184921.397080:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184921.408113:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184921.432351:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184921.443391:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:1.0:1496184921.454491:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0850 x1568861104079280 msgsize 520 00000100:00100000:1.0:1496184921.454500:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184921.454502:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184921.454521:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079280 00000020:00000040:3.0:1496184921.454525:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184921.454528:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184921.454553:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5955 00000100:00000040:3.0:1496184921.454556:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184921.454561:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0850 x1568861104079280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184941 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184921.454570:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104079280:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184921.454582:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184921.454594:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184921.454599:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184921.454602:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184921 last 1496184921 00000001:00000040:3.0:1496184921.454611:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184921.454625:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184921.454627:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184921.454635:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184921.454640:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.454642:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.454643:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184921.454644:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.454647:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.454653:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.454655:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184921.454658:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.454661:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.454666:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184921.454669:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184921.454671:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.454673:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184921.454681:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079280 00000100:00000040:3.0:1496184921.454689:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.454692:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.454711:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.454714:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0850 x1568861104079280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184941 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184921.454720:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079280:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (228us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184921.454726:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5955 00000100:00000040:3.0:1496184921.454728:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.454733:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184921.461349:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184921.472374:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184921.496724:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184921.496735:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184921.527623:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184921.527650:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184921.558836:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184921.558863:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184921.588737:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184921.588753:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184921.618989:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184921.630135:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184921.654344:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184921.665411:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184921.689873:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184921.700981:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184921.725753:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184921.736812:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184921.761249:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184921.772315:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184921.796919:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184921.807947:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184921.831802:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184921.842851:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00000040:1.0:1496184921.853966:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0c50 x1568861104079664 msgsize 520 00000100:00100000:1.0:1496184921.853974:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184921.853976:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184921.854027:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079664 00000020:00000040:3.0:1496184921.854031:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184921.854035:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184921.854066:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5956 00000100:00000040:3.0:1496184921.854068:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.854073:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0c50 x1568861104079664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184941 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184921.854096:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079664:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184921.854108:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184921.854121:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184921.854126:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184921.854129:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184921 last 1496184921 00000001:00000040:3.0:1496184921.854137:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184921.854141:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184921.854143:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184921.854152:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184921.854156:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.854158:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.854159:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184921.854160:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184921.854163:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.854169:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.854172:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184921.854175:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.854177:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184921.854183:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184921.854185:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184921.854188:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184921.854189:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184921.854198:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079664 00000100:00000040:3.0:1496184921.854205:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.854209:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.854237:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184921.854240:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0c50 x1568861104079664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184941 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184921.854246:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079664:12345-10.37.248.124@o2ib1:8 Request procesed in 158us (280us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184921.854251:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5956 00000100:00000040:3.0:1496184921.854253:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184921.854259:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184921.861397:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184921.872421:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184921.896998:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184921.897009:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184921.928388:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184921.928414:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00000100:00020000:0.0:1496184921.958863:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:0.0:1496184921.958888:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00000100:00020000:1.0:1496184921.989206:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:1.0:1496184921.989230:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00000100:00020000:1.0:1496184922.019383:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:1.0:1496184922.030541:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00000100:00000040:0.0:1496184922.043234:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f850 x1568861104079856 msgsize 520 00000100:00100000:0.0:1496184922.043243:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184922.043245:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184922.043292:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079856 00000020:00000040:3.0:1496184922.043296:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184922.043300:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184922.043328:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5957 00000100:00000040:3.0:1496184922.043330:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184922.043335:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f850 x1568861104079856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184942 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184922.043344:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079856:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184922.043356:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184922.043368:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184922.043372:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184922.043376:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184922 last 1496184921 00000001:00000040:3.0:1496184922.043385:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184922.043388:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184922.043392:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184922.043401:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184922.043405:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184922.043407:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184922.043408:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184922.043409:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184922.043412:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184922.043418:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184922.043420:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184922.043423:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184922.043426:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184922.043431:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184922.043433:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184922.043436:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184922.043438:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184922.043445:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079856 00000100:00000040:3.0:1496184922.043452:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184922.043456:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184922.043476:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184922.043479:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f850 x1568861104079856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184942 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184922.043486:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079856:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (251us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184922.043491:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5957 00000100:00000040:3.0:1496184922.043493:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184922.043498:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:1.0:1496184922.045645:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f400 00000100:00020000:1.0:1496184922.056678:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f400 00000100:00020000:3.0:1496184922.071671:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184922.082714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184922.106847:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184922.106871:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184922.137841:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184922.137865:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00000100:00000040:0.0:1496184922.166900:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0050 x1568861104079984 msgsize 520 00000100:00100000:0.0:1496184922.166909:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184922.166910:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184922.166942:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104079984 00000020:00000040:1.0:1496184922.166947:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:1.0:1496184922.166950:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:1.0:1496184922.166978:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5958 00000100:00000040:1.0:1496184922.166981:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:1.0:1496184922.166986:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0050 x1568861104079984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184942 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184922.166994:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079984:12345-10.37.248.124@o2ib1:8 00000020:00000040:1.0:1496184922.167009:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:1.0:1496184922.167021:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:1.0:1496184922.167025:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:1.0:1496184922.167029:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184922 last 1496184921 00000001:00000040:1.0:1496184922.167037:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:1.0:1496184922.167040:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:1.0:1496184922.167042:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:1.0:1496184922.167051:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:1.0:1496184922.167055:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:1.0:1496184922.167057:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:1.0:1496184922.167058:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:1.0:1496184922.167059:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:1.0:1496184922.167062:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184922.167068:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:1.0:1496184922.167070:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:1.0:1496184922.167074:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184922.167076:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:1.0:1496184922.167081:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:1.0:1496184922.167084:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:1.0:1496184922.167086:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:1.0:1496184922.167088:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:1.0:1496184922.167095:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104079984 00000100:00000040:1.0:1496184922.167103:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184922.167107:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:1.0:1496184922.167126:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184922.167128:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0050 x1568861104079984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184942 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:1.0:1496184922.167142:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104079984:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (244us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184922.167147:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5958 00000100:00000040:1.0:1496184922.167149:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:1.0:1496184922.167154:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184922.169672:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184922.169700:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184922.200619:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184922.211771:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184922.235755:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184922.246844:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184922.263050:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184922.274057:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184922.298224:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184922.298235:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184922.329510:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184922.329534:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00000100:00000040:3.0:1496184922.334086:0:6132:0:(events.c:347:request_in_callback()) incoming req@ffff8807f9348050 x1568861152397376 msgsize 224 00000100:00100000:3.0:1496184922.334094:0:6132:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:3.0:1496184922.334096:0:6132:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184922.334116:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397376 00000020:00000040:2.0:1496184922.334121:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000040:2.0:1496184922.334124:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000100:00000040:2.0:1496184922.334136:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:2.0:1496184922.334152:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 949 00000100:00000040:2.0:1496184922.334154:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000040:2.0:1496184922.334159:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f9348050 x1568861152397376/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:673/0 lens 224/0 e 0 to 0 dl 1496184928 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184922.334168:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397376:12345-10.37.248.197@o2ib1:13 00002000:00000024:2.0:1496184922.334187:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 9758656 pending 0 free 80206888960 avail 76094111744 00010000:00000040:2.0:1496184922.334196:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397376 00000100:00000040:2.0:1496184922.334204:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:2.0:1496184922.334209:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:2.0:1496184922.334229:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:2.0:1496184922.334232:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f9348050 x1568861152397376/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:673/0 lens 224/336 e 0 to 0 dl 1496184928 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:2.0:1496184922.334238:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397376:12345-10.37.248.197@o2ib1:13 Request procesed in 71us (151us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184922.334243:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 949 00000100:00000040:2.0:1496184922.334245:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000020:00000040:2.0:1496184922.334251:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 00000100:00020000:3.0:1496184922.360855:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184922.360882:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184922.391344:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184922.391368:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184922.421736:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184922.432877:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00000100:00000040:0.0:1496184922.443987:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f050 x1568861104080272 msgsize 520 00000100:00100000:0.0:1496184922.443995:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184922.443997:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184922.444021:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104080272 00000020:00000040:2.0:1496184922.444026:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184922.444030:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:2.0:1496184922.444054:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5959 00000100:00000040:2.0:1496184922.444056:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:2.0:1496184922.444061:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f050 x1568861104080272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184942 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184922.444070:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104080272:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184922.444081:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184922.444093:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184922.444098:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184922.444102:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184922 last 1496184921 00000001:00000040:2.0:1496184922.444110:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184922.444113:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184922.444115:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184922.444123:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184922.444130:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184922.444132:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184922.444133:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184922.444134:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184922.444137:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.444142:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184922.444145:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184922.444148:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.444150:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184922.444155:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184922.444158:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184922.444160:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184922.444162:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184922.444170:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104080272 00000100:00000040:2.0:1496184922.444176:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.444180:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.444198:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.444201:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f050 x1568861104080272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184942 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184922.444207:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080272:12345-10.37.248.124@o2ib1:8 Request procesed in 138us (219us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184922.444212:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5959 00000100:00000040:2.0:1496184922.444214:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184922.444219:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184922.446868:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184922.457876:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184922.486743:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184922.497784:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184922.522541:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184922.533627:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184922.558397:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184922.569465:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184922.593643:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184922.604680:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184922.628638:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184922.639689:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00000040:1.0:1496184922.650800:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8fc50 x1568861104080464 msgsize 520 00000100:00100000:1.0:1496184922.650808:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184922.650810:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184922.650860:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104080464 00000020:00000040:2.0:1496184922.650865:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184922.650869:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184922.650905:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5960 00000100:00000040:2.0:1496184922.650908:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184922.650916:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8fc50 x1568861104080464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184942 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184922.650924:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080464:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184922.650936:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184922.650948:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184922.650960:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184922.650964:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184922 last 1496184921 00000001:00000040:2.0:1496184922.650972:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184922.650976:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184922.650978:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184922.650987:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184922.650991:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184922.650993:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184922.650994:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184922.650995:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184922.650998:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.651004:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184922.651007:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184922.651010:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.651012:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184922.651018:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184922.651020:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184922.651023:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184922.651025:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184922.651041:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104080464 00000100:00000040:2.0:1496184922.651049:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.651053:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.651073:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184922.651075:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8fc50 x1568861104080464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184942 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184922.651082:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080464:12345-10.37.248.124@o2ib1:8 Request procesed in 158us (281us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184922.651087:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5960 00000100:00000040:2.0:1496184922.651089:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184922.651094:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184922.661051:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184922.672069:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184922.696634:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184922.696645:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184922.727300:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184922.727326:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184922.758247:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184922.758258:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c600 00000100:00020000:3.0:1496184922.788635:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184922.788660:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184922.819822:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184922.830978:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184922.855847:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184922.866908:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184922.891643:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184922.902726:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184922.927061:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184922.938107:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184922.962542:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184922.973589:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184922.998069:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184923.009108:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184923.033099:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184923.044153:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00000040:2.0:1496184923.055269:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ec50 x1568861104080848 msgsize 520 00000100:00100000:2.0:1496184923.055277:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184923.055279:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184923.055299:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104080848 00000020:00000040:3.0:1496184923.055303:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184923.055307:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184923.055334:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5961 00000100:00000040:3.0:1496184923.055336:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184923.055342:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ec50 x1568861104080848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184943 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184923.055351:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104080848:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184923.055363:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184923.055375:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184923.055380:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184923.055384:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184923 last 1496184923 00000001:00000040:3.0:1496184923.055392:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184923.055395:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184923.055397:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184923.055405:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184923.055410:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.055412:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.055413:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184923.055414:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184923.055417:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.055422:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.055425:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184923.055428:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.055430:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.055436:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184923.055438:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184923.055442:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.055444:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184923.055452:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104080848 00000100:00000040:3.0:1496184923.055459:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.055463:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.055482:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.055485:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ec50 x1568861104080848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184943 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184923.055491:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080848:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184923.055496:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5961 00000100:00000040:3.0:1496184923.055498:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.055504:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184923.061360:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d400 00000100:00020000:3.0:1496184923.072376:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d400 00000100:00100000:4.0:1496184923.093993:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104080880 00000020:00000040:4.0:1496184923.093996:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184923.093998:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184923.094012:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42756 00000100:00000040:4.0:1496184923.094014:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184923.094017:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2850 x1568861104080880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184943 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184923.094024:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080880:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184923.094030:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184923.094035:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184923.094038:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184923.094041:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184923 last 1496184923 00000001:00000040:4.0:1496184923.094045:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184923.094049:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184923.094050:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184923.094055:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184923.094058:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184923.094059:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184923.094060:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184923.094061:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184923.094063:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184923.094067:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184923.094069:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184923.094071:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184923.094073:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184923.094076:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184923.094079:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184923.094080:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184923.094082:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184923.094085:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104080880 00000100:00000040:4.0:1496184923.094089:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184923.094092:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184923.094104:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184923.094106:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2850 x1568861104080880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184943 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184923.094113:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104080880:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (168us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184923.094117:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42756 00000100:00000040:4.0:1496184923.094118:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184923.094122:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184923.094715:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184923.094718:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184923.094720:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184923.094725:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184923.094733:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/0 e 0 to 0 dl 1496184966 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184923.094740:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184923.094742:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:4.0:1496184923.094752:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56879 00000100:00000040:4.0:1496184923.094754:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184923.094756:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/0 e 0 to 0 dl 1496184966 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184923.094763:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184923.094805:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184923.096653:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b8c00 00000100:00020000:3.0:1496184923.096677:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b8c00 00010000:00020000:4.0:1496184923.147876:0:17513:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/448 e 0 to 0 dl 1496184966 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184923.147949:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184923.147954:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184923.147957:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/448 e 0 to 0 dl 1496184966 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184923.147969:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184923.147971:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/448 e 0 to 0 dl 1496184966 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184923.147977:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+10:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 53214us (53313us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184923.147982:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56879 00000100:00000040:4.0:1496184923.147983:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:4.0:1496184923.147988:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:711/0 lens 608/448 e 0 to 0 dl 1496184966 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184923.147994:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184923.147996:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000020:00000040:4.0:1496184923.147999:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184923.149736:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184923.160881:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184923.183556:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184923.194607:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184923.214464:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184923.225516:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184923.245579:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184923.256615:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184923.276115:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184923.287153:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184923.311511:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184923.322562:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184923.341802:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184923.352850:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184923.372384:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184923.383419:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184923.407603:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.418642:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.442084:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.453126:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.477032:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184923.488075:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184923.512052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184923.523083:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184923.547419:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184923.558485:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184923.582386:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184923.593438:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184923.617737:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184923.628774:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184923.653151:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184923.664201:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184923.688528:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184923.699559:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184923.724090:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184923.735135:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184923.759560:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184923.770586:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184923.794986:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.806028:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.829855:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.840905:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184923.852022:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e050 x1568861104081600 msgsize 520 00000100:00100000:1.0:1496184923.852031:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184923.852033:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184923.852053:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104081600 00000020:00000040:3.0:1496184923.852057:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184923.852061:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184923.852089:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5962 00000100:00000040:3.0:1496184923.852091:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184923.852097:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e050 x1568861104081600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184943 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184923.852106:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104081600:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184923.852119:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184923.852131:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184923.852136:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184923.852139:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184923 last 1496184923 00000001:00000040:3.0:1496184923.852147:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184923.852158:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184923.852161:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184923.852169:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184923.852173:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.852175:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.852176:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184923.852177:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184923.852180:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.852186:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.852188:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184923.852191:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.852194:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.852199:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184923.852202:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184923.852204:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.852206:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184923.852215:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104081600 00000100:00000040:3.0:1496184923.852222:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.852226:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.852246:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.852249:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e050 x1568861104081600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184943 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184923.852255:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104081600:12345-10.37.248.124@o2ib1:8 Request procesed in 150us (232us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184923.852261:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5962 00000100:00000040:3.0:1496184923.852263:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.852269:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184923.861255:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.872264:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.896395:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.896407:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184923.925609:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f450 x1568861104081664 msgsize 520 00000100:00100000:1.0:1496184923.925618:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184923.925619:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184923.925665:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104081664 00000020:00000040:3.0:1496184923.925669:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184923.925673:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184923.925699:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5963 00000100:00000040:3.0:1496184923.925701:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184923.925705:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f450 x1568861104081664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184943 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184923.925713:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104081664:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184923.925724:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184923.925734:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184923.925737:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184923.925741:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184923 last 1496184923 00000001:00000040:3.0:1496184923.925748:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184923.925753:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184923.925755:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184923.925762:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184923.925765:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.925767:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.925768:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184923.925769:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184923.925772:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.925777:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.925779:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184923.925783:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.925785:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184923.925789:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184923.925792:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184923.925794:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184923.925796:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184923.925803:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104081664 00000100:00000040:3.0:1496184923.925810:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.925813:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.925836:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184923.925839:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f450 x1568861104081664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184943 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184923.925845:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104081664:12345-10.37.248.124@o2ib1:8 Request procesed in 133us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184923.925850:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5963 00000100:00000040:3.0:1496184923.925852:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184923.925857:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184923.928396:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.928406:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184923.959444:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.959469:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184923.990000:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184923.990015:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184924.019832:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.030982:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.054784:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.065868:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.089504:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.100589:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.122069:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.133107:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.157725:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184924.168773:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184924.188488:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184924.199524:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184924.223548:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.234571:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184924.245687:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e850 x1568861104081984 msgsize 520 00000100:00100000:1.0:1496184924.245696:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184924.245697:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184924.245718:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104081984 00000020:00000040:3.0:1496184924.245722:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184924.245725:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184924.245752:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5964 00000100:00000040:3.0:1496184924.245755:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.245760:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e850 x1568861104081984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184944 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184924.245768:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104081984:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184924.245781:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184924.245793:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184924.245800:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184924.245804:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184924 last 1496184923 00000001:00000040:3.0:1496184924.245812:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184924.245823:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184924.245825:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184924.245834:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184924.245838:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.245840:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.245841:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184924.245842:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.245845:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.245851:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.245853:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184924.245856:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.245859:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.245864:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184924.245867:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184924.245870:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.245871:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184924.245880:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104081984 00000100:00000040:3.0:1496184924.245887:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.245899:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.245919:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.245922:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e850 x1568861104081984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184944 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184924.245928:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104081984:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184924.245935:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5964 00000100:00000040:3.0:1496184924.245937:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.245942:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184924.248526:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.259554:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.286754:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.297766:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.320994:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184924.332030:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000040:2.0:1496184924.340924:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184924.340934:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184924.340954:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.340966:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76b000] to pc [ptlrpcd_00_00:0] req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.340975:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184924.340978:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184924.340982:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.340987:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76b900] to pc [ptlrpcd_00_01:1] req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.340992:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184924.340994:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184924.340998:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.341006:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76bc00] to pc [ptlrpcd_00_02:2] req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184924.341013:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184924.341013:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.341017:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 5000 (4297363369) 00000100:00100000:1.0:1496184924.341026:0:13689:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:d86c02e9-1939-02b8-4331-d3612fcf0106:13689:1568861159710048:10.37.248.196@o2ib1:400 00000100:00100000:0.0:1496184924.341026:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:13690:1568861159710064:10.37.248.197@o2ib1:400 00000100:00000040:2.0:1496184924.341037:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184924.341044:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710080:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184924.341047:0:13689:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184924.341047:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184924.341053:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184924.341054:0:13689:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:0.0:1496184924.341054:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:2.0:1496184924.341058:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184924.343061:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184924.343074:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184924.343083:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184924.343098:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184924.343117:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184924.343126:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184924.343147:0:13690:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000040:1.0:1496184924.343148:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:3.0:1496184924.343150:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184924.343154:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:3.0:1496184924.343156:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:1.0:1496184924.343159:0:13689:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:d86c02e9-1939-02b8-4331-d3612fcf0106:13689:1568861159710048:10.37.248.196@o2ib1:400 00000100:00100000:3.0:1496184924.343161:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0000_UUID:13690:1568861159710064:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184924.343163:0:13689:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76b000 x1568861159710048/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000040:3.0:1496184924.343165:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76b900 x1568861159710064/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:1.0:1496184924.343169:0:13689:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000020:00000040:3.0:1496184924.343170:0:13690:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:3.0:1496184924.343187:0:13691:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:3.0:1496184924.343190:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:3.0:1496184924.343195:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:3.0:1496184924.343202:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710080:10.37.248.197@o2ib1:400 00000100:00000040:3.0:1496184924.343205:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76bc00 x1568861159710080/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184931 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:3.0:1496184924.343209:0:13691:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00020000:3.0:1496184924.356442:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.367473:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.387428:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.398456:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.422564:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.433594:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00000040:1.0:1496184924.444693:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d050 x1568861104082176 msgsize 520 00000100:00100000:1.0:1496184924.444702:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184924.444704:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184924.444723:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104082176 00000020:00000040:3.0:1496184924.444727:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184924.444731:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184924.444757:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5965 00000100:00000040:3.0:1496184924.444760:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.444765:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d050 x1568861104082176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184944 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184924.444774:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104082176:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184924.444786:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184924.444799:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184924.444803:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184924.444807:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184924 last 1496184923 00000001:00000040:3.0:1496184924.444822:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184924.444825:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184924.444828:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184924.444836:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184924.444843:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.444846:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.444847:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184924.444848:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.444851:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.444857:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.444859:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184924.444863:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.444865:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.444870:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184924.444873:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184924.444875:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.444877:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184924.444885:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104082176 00000100:00000040:3.0:1496184924.444901:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.444905:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.444925:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.444928:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d050 x1568861104082176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184944 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184924.444934:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104082176:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184924.444939:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5965 00000100:00000040:3.0:1496184924.444941:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.444947:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184924.447596:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.458592:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.486874:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.497883:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.521586:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.532623:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.556151:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.567206:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.591168:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184924.602201:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184924.626393:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.637435:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184924.648534:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d850 x1568861104082384 msgsize 520 00000100:00100000:1.0:1496184924.648543:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184924.648545:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184924.648565:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104082384 00000020:00000040:3.0:1496184924.648568:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184924.648572:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184924.648598:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5966 00000100:00000040:3.0:1496184924.648600:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.648605:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d850 x1568861104082384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184944 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184924.648614:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104082384:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184924.648627:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184924.648640:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184924.648644:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184924.648648:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184924 last 1496184923 00000001:00000040:3.0:1496184924.648664:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184924.648667:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184924.648669:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184924.648678:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184924.648682:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.648684:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.648685:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184924.648686:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184924.648692:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.648698:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.648701:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184924.648704:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.648706:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184924.648711:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184924.648714:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184924.648716:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184924.648718:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184924.648726:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104082384 00000100:00000040:3.0:1496184924.648734:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.648737:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.648758:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184924.648761:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d850 x1568861104082384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184944 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184924.648768:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104082384:12345-10.37.248.124@o2ib1:8 Request procesed in 154us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184924.648773:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5966 00000100:00000040:3.0:1496184924.648775:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184924.648781:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184924.661438:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.672433:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184924.696270:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184924.696282:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184924.727125:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.727149:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184924.757256:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.757279:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184924.788274:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.788288:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184924.819051:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.830212:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184924.854717:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184924.865795:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184924.890177:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.901261:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184924.925618:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184924.936666:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184924.961290:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184924.972352:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184924.996825:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184925.007860:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184925.031862:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184925.042912:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184925.053975:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c850 x1568861104082768 msgsize 520 00000100:00100000:1.0:1496184925.053984:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184925.053986:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.054016:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104082768 00000020:00000040:3.0:1496184925.054020:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184925.054023:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184925.054049:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5967 00000100:00000040:3.0:1496184925.054051:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184925.054056:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c850 x1568861104082768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184945 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.054065:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104082768:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184925.054078:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184925.054090:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184925.054094:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184925.054098:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184925 last 1496184925 00000001:00000040:3.0:1496184925.054106:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184925.054109:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184925.054114:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184925.054122:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184925.054127:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.054129:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.054130:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184925.054131:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.054134:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.054140:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.054142:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184925.054145:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.054148:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.054160:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184925.054163:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184925.054166:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.054167:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184925.054175:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104082768 00000100:00000040:3.0:1496184925.054183:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.054187:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.054207:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.054209:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c850 x1568861104082768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184945 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184925.054216:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104082768:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (239us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.054221:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5967 00000100:00000040:3.0:1496184925.054223:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.054229:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184925.061150:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184925.072185:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184925.096761:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184925.096785:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184925.127160:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184925.127171:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184925.157925:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184925.157936:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184925.188695:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184925.188706:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184925.218950:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184925.230099:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184925.254571:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184925.265664:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184925.286814:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184925.297900:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184925.321617:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184925.332680:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00000040:2.0:1496184925.349759:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e450 x1568861104083056 msgsize 520 00000100:00100000:2.0:1496184925.349768:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184925.349770:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.349817:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083056 00000020:00000040:3.0:1496184925.349821:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184925.349825:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184925.349852:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5968 00000100:00000040:3.0:1496184925.349855:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.349860:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e450 x1568861104083056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184945 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.349869:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083056:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184925.349881:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184925.349906:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184925.349912:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184925.349915:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184925 last 1496184925 00000001:00000040:3.0:1496184925.349928:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184925.349931:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184925.349933:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184925.349942:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184925.349946:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.349949:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.349950:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184925.349951:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.349954:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.349960:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.349962:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184925.349966:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.349968:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.349973:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184925.349976:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184925.349978:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.349980:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184925.349988:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083056 00000100:00000040:3.0:1496184925.349996:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.349999:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.350019:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.350022:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e450 x1568861104083056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184945 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184925.350028:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083056:12345-10.37.248.124@o2ib1:8 Request procesed in 160us (268us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.350033:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5968 00000100:00000040:3.0:1496184925.350036:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.350042:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184925.352431:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184925.363481:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184925.387746:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184925.398794:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184925.422525:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184925.433580:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:1.0:1496184925.444695:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d450 x1568861104083152 msgsize 520 00000100:00100000:1.0:1496184925.444704:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184925.444706:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.444726:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083152 00000020:00000040:3.0:1496184925.444729:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184925.444733:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184925.444758:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5969 00000100:00000040:3.0:1496184925.444761:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184925.444767:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d450 x1568861104083152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184945 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.444775:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104083152:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184925.444788:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184925.444800:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184925.444805:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184925.444808:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184925 last 1496184925 00000001:00000040:3.0:1496184925.444816:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184925.444827:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184925.444829:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184925.444838:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184925.444842:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.444844:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.444845:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184925.444847:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.444852:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.444858:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.444860:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184925.444863:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.444866:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.444871:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184925.444874:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184925.444877:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.444878:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184925.444886:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083152 00000100:00000040:3.0:1496184925.444902:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.444906:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.444926:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.444929:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d450 x1568861104083152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184945 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184925.444935:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083152:12345-10.37.248.124@o2ib1:8 Request procesed in 160us (239us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.444941:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5969 00000100:00000040:3.0:1496184925.444943:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.444949:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184925.447466:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184925.458495:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184925.486995:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184925.498028:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184925.522530:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184925.533580:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184925.557189:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184925.568240:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184925.592265:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184925.603321:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184925.627533:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184925.638572:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:1.0:1496184925.649672:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8dc50 x1568861104083344 msgsize 520 00000100:00100000:1.0:1496184925.649680:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184925.649682:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.649702:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083344 00000020:00000040:3.0:1496184925.649706:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184925.649709:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184925.649735:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5970 00000100:00000040:3.0:1496184925.649738:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184925.649743:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8dc50 x1568861104083344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184945 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.649752:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104083344:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184925.649764:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184925.649777:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184925.649782:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184925.649785:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184925 last 1496184925 00000001:00000040:3.0:1496184925.649793:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184925.649796:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184925.649799:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184925.649807:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184925.649811:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.649813:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.649814:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184925.649815:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.649826:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.649831:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.649834:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184925.649837:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.649842:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.649847:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184925.649850:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184925.649852:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.649854:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184925.649862:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083344 00000100:00000040:3.0:1496184925.649870:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.649873:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.649902:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.649905:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8dc50 x1568861104083344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184945 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184925.649911:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083344:12345-10.37.248.124@o2ib1:8 Request procesed in 160us (239us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.649916:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5970 00000100:00000040:3.0:1496184925.649919:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.649925:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184925.661518:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184925.672524:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184925.696489:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184925.696499:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184925.728067:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184925.728092:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184925.758600:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184925.758623:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184925.789119:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184925.789145:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184925.819946:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184925.831096:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:0.0:1496184925.842205:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8bc50 x1568861104083536 msgsize 520 00000100:00100000:0.0:1496184925.842222:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184925.842224:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.842244:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083536 00000020:00000040:3.0:1496184925.842250:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184925.842253:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184925.842278:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5971 00000100:00000040:3.0:1496184925.842280:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184925.842285:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8bc50 x1568861104083536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184945 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.842295:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104083536:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184925.842307:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184925.842319:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184925.842323:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184925.842327:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184925 last 1496184925 00000001:00000040:3.0:1496184925.842335:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184925.842338:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184925.842340:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184925.842348:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184925.842352:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.842362:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.842363:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184925.842364:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184925.842368:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.842373:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.842376:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184925.842379:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.842381:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184925.842387:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184925.842389:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184925.842393:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184925.842395:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184925.842403:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083536 00000100:00000040:3.0:1496184925.842411:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.842414:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.842443:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184925.842445:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8bc50 x1568861104083536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184945 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184925.842451:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083536:12345-10.37.248.124@o2ib1:8 Request procesed in 158us (246us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.842457:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5971 00000100:00000040:3.0:1496184925.842459:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184925.842464:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184925.845017:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184925.856047:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184925.885443:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184925.896499:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:1.0:1496184925.907544:0:6132:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee5450 x1568861152397424 msgsize 224 00000100:00100000:1.0:1496184925.907552:0:6132:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:1.0:1496184925.907553:0:6132:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184925.907568:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397424 00000020:00000040:3.0:1496184925.907579:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:3.0:1496184925.907581:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:3.0:1496184925.907589:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:3.0:1496184925.907602:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 950 00000100:00000040:3.0:1496184925.907604:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:3.0:1496184925.907607:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee5450 x1568861152397424/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:676/0 lens 224/0 e 0 to 0 dl 1496184931 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184925.907615:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397424:12345-10.37.248.197@o2ib1:13 00002000:00000024:3.0:1496184925.907632:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:3.0:1496184925.907637:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397424 00000100:00000040:3.0:1496184925.907642:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:3.0:1496184925.907648:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184925.907663:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:3.0:1496184925.907670:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee5450 x1568861152397424/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:676/0 lens 224/336 e 0 to 0 dl 1496184931 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:3.0:1496184925.907676:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397424:12345-10.37.248.197@o2ib1:13 Request procesed in 62us (131us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184925.907681:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 950 00000100:00000040:3.0:1496184925.907683:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:3.0:1496184925.907687:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00000100:00020000:3.0:1496184925.920985:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184925.932041:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184925.955999:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184925.967068:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:0.0:1496184925.991555:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:0.0:1496184926.002622:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00010000:00020000:4.0:1496184926.013975:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:713/0 lens 608/448 e 0 to 0 dl 1496184968 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184926.014022:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184926.014027:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184926.014031:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:713/0 lens 608/448 e 0 to 0 dl 1496184968 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184926.014035:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184926.014038:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:713/0 lens 608/448 e 0 to 0 dl 1496184968 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184926.014044:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24306us (24401us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184926.014049:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56965 00000100:00000040:4.0:1496184926.014051:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184926.014054:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:713/0 lens 608/448 e 0 to 0 dl 1496184968 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184926.014060:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.014063:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000020:00000040:4.0:1496184926.014066:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:1.0:1496184926.015595:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c450 x1568861104083696 msgsize 520 00000100:00100000:1.0:1496184926.015604:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184926.015608:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184926.015658:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083696 00000020:00000040:3.0:1496184926.015663:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184926.015666:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184926.015693:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5972 00000100:00000040:3.0:1496184926.015696:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184926.015701:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c450 x1568861104083696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184946 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184926.015709:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083696:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184926.015722:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184926.015733:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184926.015738:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184926.015741:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184926 last 1496184925 00000001:00000040:3.0:1496184926.015749:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184926.015752:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184926.015755:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184926.015765:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184926.015770:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184926.015772:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184926.015773:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184926.015774:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184926.015777:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.015783:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184926.015785:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184926.015788:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.015790:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184926.015795:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184926.015798:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184926.015800:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184926.015802:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184926.015810:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083696 00000100:00000040:3.0:1496184926.015818:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.015821:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.015841:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.015843:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c450 x1568861104083696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184946 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184926.015849:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083696:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (253us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184926.015855:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5972 00000100:00000040:3.0:1496184926.015856:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184926.015862:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184926.016414:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184926.016417:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184926.016419:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184926.016427:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184926.016434:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184926.016441:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.016443:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000100:00100000:4.0:1496184926.016453:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56966 00000100:00000040:4.0:1496184926.016455:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184926.016457:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184926.016464:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184926.016510:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184926.018015:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b8a00 00000100:00020000:0.0:1496184926.029053:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b8a00 00010000:00020000:4.0:1496184926.040147:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184926.040206:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184926.040211:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184926.040214:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184926.040219:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184926.040221:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184926.040227:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23763us (23859us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184926.040232:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56966 00000100:00000040:4.0:1496184926.040234:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184926.040238:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184926.040250:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.040252:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000020:00000040:4.0:1496184926.040256:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184926.042284:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184926.042287:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184926.042288:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184926.042294:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184926.042302:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184926.042308:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.042311:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000100:00100000:4.0:1496184926.042320:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56967 00000100:00000040:4.0:1496184926.042322:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184926.042324:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184926.042331:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184926.042372:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:1.0:1496184926.043844:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9400 00000100:00020000:1.0:1496184926.054888:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9400 00010000:00020000:4.0:1496184926.065982:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184926.066028:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184926.066033:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184926.066036:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184926.066041:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184926.066044:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184926.066056:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+10:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23725us (23817us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184926.066061:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56967 00000100:00000040:4.0:1496184926.066063:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:4.0:1496184926.066066:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184926.066072:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.066075:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000020:00000040:4.0:1496184926.066078:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184926.067133:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184926.067136:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184926.067138:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184926.067143:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184926.067151:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184926.067158:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.067160:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000100:00100000:4.0:1496184926.067170:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 56968 00000100:00000040:4.0:1496184926.067172:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184926.067174:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/0 e 0 to 0 dl 1496184969 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184926.067181:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184926.067221:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184926.070173:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9400 00000100:00020000:3.0:1496184926.081191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9400 00010000:00020000:4.0:1496184926.092234:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184926.092292:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184926.092304:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184926.092307:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184926.092312:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184926.092315:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184926.092321:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 25140us (25235us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184926.092326:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 56968 00000100:00000040:4.0:1496184926.092327:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184926.092331:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:714/0 lens 608/448 e 0 to 0 dl 1496184969 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184926.092336:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184926.092339:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000020:00000040:4.0:1496184926.092342:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184926.105164:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184926.105176:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184926.136045:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184926.136056:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184926.167201:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184926.167208:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184926.197965:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184926.209133:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184926.232996:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.244095:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:1.0:1496184926.255200:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c050 x1568861104083952 msgsize 520 00000100:00100000:1.0:1496184926.255209:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184926.255211:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184926.255232:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104083952 00000020:00000040:3.0:1496184926.255244:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184926.255248:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184926.255274:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5973 00000100:00000040:3.0:1496184926.255276:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184926.255281:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c050 x1568861104083952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184946 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184926.255290:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104083952:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184926.255302:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184926.255314:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184926.255319:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184926.255323:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184926 last 1496184925 00000001:00000040:3.0:1496184926.255331:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184926.255334:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184926.255337:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184926.255345:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184926.255350:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184926.255352:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184926.255356:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184926.255357:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184926.255360:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.255366:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184926.255368:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184926.255372:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.255381:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184926.255387:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184926.255389:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184926.255392:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184926.255394:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184926.255402:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104083952 00000100:00000040:3.0:1496184926.255409:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.255413:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.255434:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184926.255436:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c050 x1568861104083952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184946 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184926.255442:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104083952:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184926.255448:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5973 00000100:00000040:3.0:1496184926.255450:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184926.255456:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184926.258308:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.269334:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.293592:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184926.304636:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184926.328758:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184926.339847:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184926.363559:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184926.374608:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184926.398708:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184926.409744:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184926.433852:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.444909:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:1.0:1496184926.456053:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8cc50 x1568861104084144 msgsize 520 00000100:00100000:1.0:1496184926.456062:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184926.456063:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184926.456084:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084144 00000020:00000040:2.0:1496184926.456089:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184926.456092:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184926.456120:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5974 00000100:00000040:2.0:1496184926.456122:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184926.456128:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8cc50 x1568861104084144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184946 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184926.456137:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084144:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184926.456149:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184926.456162:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184926.456167:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184926.456170:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184926 last 1496184925 00000001:00000040:2.0:1496184926.456179:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184926.456182:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184926.456184:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184926.456194:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184926.456198:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.456200:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.456201:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184926.456202:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184926.456205:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.456211:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.456213:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184926.456216:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.456218:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.456223:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184926.456226:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184926.456228:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.456230:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184926.456238:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084144 00000100:00000040:2.0:1496184926.456245:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.456248:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.456269:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.456271:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8cc50 x1568861104084144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184946 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184926.456278:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084144:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (224us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184926.456283:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5974 00000100:00000040:2.0:1496184926.456285:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.456290:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184926.458816:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.469841:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184926.494198:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184926.505256:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184926.530062:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184926.541120:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000040:4.0:1496184926.563056:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd450 x1568861104084240 msgsize 520 00000100:00100000:4.0:1496184926.563063:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184926.563064:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184926.563754:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184926.563761:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184926.563762:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184926.565519:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184926.576588:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:4.0:1496184926.598741:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184926.598748:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184926.598750:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184926.600458:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184926.611497:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:4.0:1496184926.633340:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bddc50 x1568861104084304 msgsize 520 00000100:00100000:4.0:1496184926.633346:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184926.633348:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184926.635799:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184926.646846:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:1.0:1496184926.657958:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b450 x1568861104084336 msgsize 520 00000100:00100000:1.0:1496184926.657966:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184926.657968:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184926.658016:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084336 00000020:00000040:2.0:1496184926.658021:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184926.658024:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184926.658054:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5975 00000100:00000040:2.0:1496184926.658056:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184926.658061:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b450 x1568861104084336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184946 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184926.658070:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084336:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184926.658082:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184926.658094:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184926.658099:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184926.658102:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184926 last 1496184925 00000001:00000040:2.0:1496184926.658110:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184926.658114:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184926.658116:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184926.658124:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184926.658128:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.658131:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.658132:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184926.658133:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184926.658136:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.658141:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.658144:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184926.658147:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.658149:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.658154:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184926.658157:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184926.658160:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.658161:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184926.658170:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084336 00000100:00000040:2.0:1496184926.658178:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.658181:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.658201:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.658203:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b450 x1568861104084336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184946 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184926.658209:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084336:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (251us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184926.658215:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5975 00000100:00000040:2.0:1496184926.658216:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.658221:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184926.659094:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184926.659101:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184926.659103:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184926.670388:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184926.681400:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184926.706037:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184926.706048:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184926.737658:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184926.737683:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184926.768552:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184926.768579:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184926.799343:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184926.810496:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184926.834397:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184926.845459:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00000100:00000040:1.0:1496184926.856600:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a050 x1568861104084528 msgsize 520 00000100:00100000:1.0:1496184926.856609:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184926.856610:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184926.856631:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084528 00000020:00000040:2.0:1496184926.856635:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184926.856639:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:2.0:1496184926.856664:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5976 00000100:00000040:2.0:1496184926.856667:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:2.0:1496184926.856672:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a050 x1568861104084528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184946 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184926.856681:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104084528:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184926.856694:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184926.856706:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184926.856711:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184926.856714:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184926 last 1496184925 00000001:00000040:2.0:1496184926.856722:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184926.856725:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184926.856728:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184926.856736:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184926.856742:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.856744:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.856745:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184926.856746:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184926.856749:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.856755:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.856758:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184926.856761:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.856763:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184926.856768:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184926.856771:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184926.856773:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184926.856775:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184926.856783:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084528 00000100:00000040:2.0:1496184926.856790:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.856794:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.856812:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184926.856815:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a050 x1568861104084528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184946 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184926.856822:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084528:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (221us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184926.856827:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5976 00000100:00000040:2.0:1496184926.856829:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184926.856834:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184926.859374:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184926.870388:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184926.895246:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184926.906281:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184926.931001:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184926.942062:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184926.966226:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184926.977261:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184927.001937:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184927.012960:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184927.037167:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184927.048197:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00000040:1.0:1496184927.059318:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a850 x1568861104084720 msgsize 520 00000100:00100000:1.0:1496184927.059326:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.059328:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184927.059357:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084720 00000020:00000040:2.0:1496184927.059362:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184927.059366:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184927.059394:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5977 00000100:00000040:2.0:1496184927.059396:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.059403:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a850 x1568861104084720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184927.059412:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084720:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184927.059425:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184927.059437:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184927.059441:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184927.059445:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:2.0:1496184927.059454:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184927.059457:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184927.059459:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184927.059467:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184927.059471:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.059473:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.059474:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184927.059475:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.059478:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.059484:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.059487:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184927.059490:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.059492:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.059498:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184927.059501:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184927.059503:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.059504:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184927.059513:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084720 00000100:00000040:2.0:1496184927.059520:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.059524:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.059543:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.059546:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a850 x1568861104084720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184927.059552:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084720:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (234us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184927.059557:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5977 00000100:00000040:2.0:1496184927.059559:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.059565:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184927.070350:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184927.081357:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00000040:1.0:1496184927.103028:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a450 x1568861104084752 msgsize 520 00000100:00100000:1.0:1496184927.103036:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.103037:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184927.103083:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084752 00000020:00000040:3.0:1496184927.103087:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184927.103090:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184927.103110:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5978 00000100:00000040:3.0:1496184927.103112:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184927.103116:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a450 x1568861104084752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184927.103124:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084752:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184927.103132:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184927.103139:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184927.103143:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184927.103145:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:3.0:1496184927.103151:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184927.103153:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184927.103154:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184927.103160:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184927.103166:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184927.103168:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184927.103169:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184927.103170:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184927.103172:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.103176:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184927.103178:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184927.103181:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.103183:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184927.103187:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184927.103189:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184927.103191:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184927.103193:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184927.103198:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084752 00000100:00000040:3.0:1496184927.103204:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.103207:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.103222:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.103225:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a450 x1568861104084752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184927.103231:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084752:12345-10.37.248.124@o2ib1:8 Request procesed in 107us (202us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184927.103235:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5978 00000100:00000040:3.0:1496184927.103237:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184927.103242:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184927.105634:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184927.105645:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184927.136238:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184927.136264:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d800 00000100:00020000:3.0:1496184927.167670:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184927.167694:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184927.198225:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184927.209374:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184927.233392:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184927.244442:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00000100:00000040:1.0:1496184927.255562:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b850 x1568861104084912 msgsize 520 00000100:00100000:1.0:1496184927.255571:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.255573:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184927.255592:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084912 00000020:00000040:3.0:1496184927.255596:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184927.255600:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184927.255625:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5979 00000100:00000040:3.0:1496184927.255628:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184927.255633:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b850 x1568861104084912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184927.255641:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084912:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184927.255654:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184927.255666:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184927.255671:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184927.255674:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:3.0:1496184927.255682:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184927.255685:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184927.255687:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184927.255695:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184927.255700:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184927.255702:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184927.255703:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184927.255704:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184927.255707:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.255715:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184927.255717:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184927.255720:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.255723:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184927.255728:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184927.255731:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184927.255733:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184927.255735:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184927.255743:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084912 00000100:00000040:3.0:1496184927.255750:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.255754:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.255774:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184927.255777:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b850 x1568861104084912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184927.255783:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084912:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184927.255788:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5979 00000100:00000040:3.0:1496184927.255790:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184927.255795:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.256767:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.256770:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.256772:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.256777:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.256785:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.256792:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.256794:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000100:00100000:4.0:1496184927.256804:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57004 00000100:00000040:4.0:1496184927.256806:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.256808:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.256815:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.256859:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.258389:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba800 00000100:00020000:3.0:1496184927.269403:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba800 00010000:00020000:4.0:1496184927.280459:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.280504:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.280509:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.280512:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.280517:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.280520:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.280526:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23711us (23805us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.280533:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57004 00000100:00000040:4.0:1496184927.280535:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.280538:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.280544:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.280553:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000020:00000040:4.0:1496184927.280556:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.291582:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084944 00000020:00000040:4.0:1496184927.291585:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.291587:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.291602:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42864 00000100:00000040:4.0:1496184927.291603:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.291606:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7850 x1568861104084944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.291613:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084944:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.291619:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.291625:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.291627:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.291630:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.291634:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.291636:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.291637:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.291642:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.291645:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.291646:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.291649:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.291650:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.291653:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.291657:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.291659:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.291661:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.291663:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.291666:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.291668:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.291670:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.291671:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.291675:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084944 00000100:00000040:4.0:1496184927.291679:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.291682:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.291693:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.291696:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7850 x1568861104084944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.291702:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084944:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (164us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.291706:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42864 00000100:00000040:4.0:1496184927.291708:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.291711:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.292217:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.292220:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.292221:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.292228:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.292235:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.292242:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.292246:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000100:00100000:4.0:1496184927.292256:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57005 00000100:00000040:4.0:1496184927.292258:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.292260:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.292267:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.292311:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.293833:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba400 00000100:00020000:3.0:1496184927.304860:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba400 00010000:00020000:4.0:1496184927.315930:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.315975:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.315980:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.315983:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.315988:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.315991:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.315996:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23730us (23827us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.316001:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57005 00000100:00000040:4.0:1496184927.316003:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.316007:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.316012:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.316015:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000020:00000040:4.0:1496184927.316018:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.326799:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104084976 00000020:00000040:4.0:1496184927.326804:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.326806:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.326820:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42865 00000100:00000040:4.0:1496184927.326822:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.326825:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6850 x1568861104084976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.326832:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084976:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.326838:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.326843:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.326845:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.326848:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.326852:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.326854:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.326855:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.326859:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.326862:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.326864:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.326865:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.326865:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.326867:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.326871:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.326873:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.326875:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.326877:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.326879:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.326882:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.326884:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.326885:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.326889:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104084976 00000100:00000040:4.0:1496184927.326905:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.326908:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.326919:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.326922:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6850 x1568861104084976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.326928:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104084976:12345-10.37.248.124@o2ib1:8 Request procesed in 97us (176us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.326933:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42865 00000100:00000040:4.0:1496184927.326935:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.326938:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.327518:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.327521:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.327522:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.327528:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.327535:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.327542:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.327544:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:4.0:1496184927.327554:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57006 00000100:00000040:4.0:1496184927.327556:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.327558:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.327565:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.327610:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.329062:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bbe00 00000100:00020000:3.0:1496184927.340114:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bbe00 00010000:00020000:4.0:1496184927.351174:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184927.351175:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee5850 x1568861152397440 msgsize 224 00000100:00100000:1.0:1496184927.351192:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:1.0:1496184927.351194:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184927.351213:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397440 00000020:00000040:3.0:1496184927.351217:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000040:3.0:1496184927.351220:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000100:00000040:3.0:1496184927.351231:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000020:00000040:4.0:1496184927.351234:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.351239:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.351242:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.351247:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00100000:3.0:1496184927.351248:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 951 00000100:00000040:4.0:1496184927.351250:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184927.351250:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000040:3.0:1496184927.351255:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee5850 x1568861152397440/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:678/0 lens 224/0 e 0 to 0 dl 1496184933 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.351256:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23691us (23783us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.351261:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57006 00000100:00000040:4.0:1496184927.351263:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:3.0:1496184927.351264:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397440:12345-10.37.248.197@o2ib1:13 00002000:00100000:4.0:1496184927.351267:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.351273:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.351275:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000020:00000040:4.0:1496184927.351279:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00002000:00000024:3.0:1496184927.351283:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 9758656 pending 0 free 80206888960 avail 76094111744 00010000:00000040:3.0:1496184927.351290:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397440 00000100:00000040:3.0:1496184927.351298:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:3.0:1496184927.351303:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184927.351324:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:3.0:1496184927.351326:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee5850 x1568861152397440/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:678/0 lens 224/336 e 0 to 0 dl 1496184933 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:3.0:1496184927.351337:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397440:12345-10.37.248.197@o2ib1:13 Request procesed in 74us (161us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184927.351342:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 951 00000100:00000040:3.0:1496184927.351344:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000020:00000040:3.0:1496184927.351351:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 00000100:00100000:4.0:1496184927.362132:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085008 00000020:00000040:4.0:1496184927.362135:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.362137:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.362152:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42866 00000100:00000040:4.0:1496184927.362153:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.362156:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6050 x1568861104085008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.362163:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085008:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.362169:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.362176:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.362179:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.362182:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.362186:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.362187:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.362189:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.362193:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.362196:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.362197:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.362198:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.362199:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.362201:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.362205:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.362207:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.362209:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.362211:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.362214:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.362216:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.362218:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.362219:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.362222:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085008 00000100:00000040:4.0:1496184927.362227:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.362229:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.362241:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.362243:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6050 x1568861104085008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.362250:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085008:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (165us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.362255:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42866 00000100:00000040:4.0:1496184927.362256:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.362260:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.362868:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.362871:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.362873:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.362878:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.362886:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.362906:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.362908:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:4.0:1496184927.362918:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57007 00000100:00000040:4.0:1496184927.362920:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.362922:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.362929:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.362972:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.364614:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bba00 00000100:00020000:3.0:1496184927.375654:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bba00 00010000:00020000:4.0:1496184927.386714:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.386771:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.386776:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.386779:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.386784:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.386787:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.386799:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23870us (23980us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.386804:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57007 00000100:00000040:4.0:1496184927.386805:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.386809:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.386815:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.386817:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000020:00000040:4.0:1496184927.386821:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.394640:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085040 00000020:00000040:4.0:1496184927.394643:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.394644:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.394659:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42867 00000100:00000040:4.0:1496184927.394660:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.394663:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5c50 x1568861104085040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.394670:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085040:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.394676:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.394681:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.394684:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.394687:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.394690:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.394692:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.394693:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.394698:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.394703:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.394704:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.394705:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.394706:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.394708:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.394712:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.394713:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.394716:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.394718:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.394720:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.394723:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.394725:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.394726:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.394729:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085040 00000100:00000040:4.0:1496184927.394733:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.394736:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.394748:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.394750:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5c50 x1568861104085040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.394756:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085040:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (162us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.394761:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42867 00000100:00000040:4.0:1496184927.394762:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.394766:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.395391:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.395394:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.395396:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.395402:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.395410:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.395418:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.395421:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000100:00100000:4.0:1496184927.395430:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57008 00000100:00000040:4.0:1496184927.395432:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.395434:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.395441:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.395485:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.397244:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9e00 00000100:00020000:3.0:1496184927.408287:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9e00 00010000:00020000:4.0:1496184927.419320:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.419365:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.419370:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.419373:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.419378:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.419381:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.419387:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23946us (24046us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.419392:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57008 00000100:00000040:4.0:1496184927.419393:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.419397:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.419403:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.419407:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000020:00000040:4.0:1496184927.419411:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:3.0:1496184927.429709:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b050 x1568861104085072 msgsize 520 00000100:00100000:3.0:1496184927.429717:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184927.429719:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184927.429769:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085072 00000020:00000040:0.0:1496184927.429774:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184927.429778:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184927.429806:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5980 00000100:00000040:0.0:1496184927.429809:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184927.429814:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b050 x1568861104085072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184927.429822:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085072:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184927.429835:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184927.429848:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184927.429852:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184927.429856:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:0.0:1496184927.429864:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184927.429867:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184927.429870:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184927.429878:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184927.429883:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184927.429885:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184927.429886:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184927.429887:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184927.429905:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184927.429911:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184927.429913:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184927.429916:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184927.429921:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184927.429926:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184927.429929:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184927.429932:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184927.429933:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184927.429942:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085072 00000100:00000040:0.0:1496184927.429949:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184927.429953:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184927.429974:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184927.429977:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b050 x1568861104085072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184927.429983:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085072:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (273us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184927.429988:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5980 00000100:00000040:0.0:1496184927.429990:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184927.429995:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.430524:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.430527:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.430529:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.430535:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.430543:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.430550:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.430552:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000100:00100000:4.0:1496184927.430562:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57009 00000100:00000040:4.0:1496184927.430564:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.430566:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.430572:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.430615:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.432199:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9000 00000100:00020000:3.0:1496184927.443253:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9000 00010000:00020000:4.0:1496184927.454336:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184927.454352:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89450 x1568861104085104 msgsize 520 00000100:00100000:1.0:1496184927.454359:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.454361:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184927.454383:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085104 00000020:00000040:2.0:1496184927.454386:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184927.454388:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:4.0:1496184927.454394:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184927.454399:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.454402:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184927.454406:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5981 00010000:00000040:4.0:1496184927.454407:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:2.0:1496184927.454408:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:4.0:1496184927.454410:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:2.0:1496184927.454411:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89450 x1568861104085104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.454416:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23843us (23938us total) trans 0 rc -110/-999 00000100:00100000:2.0:1496184927.454419:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104085104:12345-10.37.248.124@o2ib1:8 00000100:00100000:4.0:1496184927.454426:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57009 00000020:00000040:2.0:1496184927.454427:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000040:4.0:1496184927.454428:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:4.0:1496184927.454432:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184927.454435:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00002000:00100000:4.0:1496184927.454438:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00010000:02000400:2.0:1496184927.454439:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00002000:00010000:4.0:1496184927.454440:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00010000:00080000:2.0:1496184927.454442:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000020:00000040:4.0:1496184927.454444:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000001:00000040:2.0:1496184927.454447:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184927.454449:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184927.454451:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184927.454457:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184927.454460:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.454462:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.454465:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184927.454466:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.454469:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.454474:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.454476:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184927.454479:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.454481:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.454484:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184927.454486:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184927.454488:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.454490:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184927.454495:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085104 00000100:00000040:2.0:1496184927.454501:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.454504:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.454518:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.454521:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89450 x1568861104085104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184927.454527:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085104:12345-10.37.248.124@o2ib1:8 Request procesed in 108us (174us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184927.454532:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5981 00000100:00000040:2.0:1496184927.454534:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.454538:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.455562:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.455565:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.455567:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.455573:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.455581:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.455587:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.455590:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000100:00100000:4.0:1496184927.455599:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57010 00000100:00000040:4.0:1496184927.455601:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.455612:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.455619:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.455662:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.457109:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba600 00000100:00020000:3.0:1496184927.468117:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba600 00010000:00020000:4.0:1496184927.479156:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.479214:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.479219:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.479222:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.479229:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.479232:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.479238:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23619us (23720us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.479243:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57010 00000100:00000040:4.0:1496184927.479245:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.479248:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.479254:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.479256:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000020:00000040:4.0:1496184927.479260:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.493550:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085136 00000020:00000040:4.0:1496184927.493553:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.493555:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.493569:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42868 00000100:00000040:4.0:1496184927.493571:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.493573:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5850 x1568861104085136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.493581:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085136:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.493587:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.493592:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.493595:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.493597:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.493602:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.493606:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.493607:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.493612:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.493615:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.493616:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.493617:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.493618:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.493620:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.493625:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.493626:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.493629:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.493631:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.493633:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.493636:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.493638:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.493639:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.493643:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085136 00000100:00000040:4.0:1496184927.493647:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.493649:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.493661:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.493664:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5850 x1568861104085136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.493670:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085136:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (169us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.493674:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42868 00000100:00000040:4.0:1496184927.493676:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.493679:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.494214:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.494217:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.494218:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.494223:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.494231:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.494238:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.494240:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:4.0:1496184927.494250:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57011 00000100:00000040:4.0:1496184927.494252:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.494254:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.494260:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.494305:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.495911:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb600 00000100:00020000:3.0:1496184927.506962:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb600 00010000:00020000:4.0:1496184927.518025:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.518070:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.518075:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.518078:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.518083:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.518086:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.518091:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23831us (23922us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.518096:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57011 00000100:00000040:4.0:1496184927.518098:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.518104:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.518109:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.518112:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:4.0:1496184927.518115:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.528646:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085168 00000020:00000040:4.0:1496184927.528649:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.528651:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.528665:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42869 00000100:00000040:4.0:1496184927.528667:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.528669:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6450 x1568861104085168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.528677:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085168:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.528683:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.528688:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.528690:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.528693:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.528697:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.528698:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.528700:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.528704:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.528707:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.528708:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.528709:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.528710:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.528712:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.528718:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.528720:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.528722:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.528724:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.528726:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.528729:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.528731:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.528732:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.528735:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085168 00000100:00000040:4.0:1496184927.528740:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.528742:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.528754:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.528756:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6450 x1568861104085168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.528762:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085168:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (163us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.528767:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42869 00000100:00000040:4.0:1496184927.528768:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.528772:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.529386:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.529389:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.529391:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.529396:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.529404:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.529411:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.529413:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:4.0:1496184927.529423:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57012 00000100:00000040:4.0:1496184927.529427:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.529429:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.529436:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.529479:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.531149:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9400 00000100:00020000:3.0:1496184927.542209:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9400 00010000:00020000:4.0:1496184927.553281:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.553340:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.553344:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.553347:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.553352:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.553355:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.553361:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23925us (24019us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.553366:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57012 00000100:00000040:4.0:1496184927.553368:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.553371:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.553377:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.553379:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000020:00000040:4.0:1496184927.553383:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.564008:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085200 00000020:00000040:4.0:1496184927.564011:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.564013:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.564029:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42870 00000100:00000040:4.0:1496184927.564031:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.564034:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8450 x1568861104085200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.564041:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085200:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.564047:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.564052:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.564054:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.564057:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.564061:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.564063:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.564064:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.564069:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.564071:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.564073:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.564074:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.564075:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.564077:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.564081:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.564082:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.564084:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.564086:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.564089:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.564091:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.564093:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.564094:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.564098:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085200 00000100:00000040:4.0:1496184927.564103:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.564105:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.564117:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.564120:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8450 x1568861104085200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.564126:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085200:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (165us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.564130:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42870 00000100:00000040:4.0:1496184927.564132:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.564135:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.564716:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.564719:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.564720:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.564726:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.564734:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.564740:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.564743:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:4.0:1496184927.564753:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57013 00000100:00000040:4.0:1496184927.564754:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.564757:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.564763:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.564808:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.566203:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bac00 00000100:00020000:3.0:1496184927.577243:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bac00 00010000:00020000:4.0:1496184927.588299:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.588358:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.588365:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.588368:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.588373:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.588375:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.588381:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23618us (23708us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.588386:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57013 00000100:00000040:4.0:1496184927.588388:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.588392:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.588397:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.588400:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:4.0:1496184927.588403:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.598984:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085232 00000020:00000040:4.0:1496184927.598989:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.598991:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.599012:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42871 00000100:00000040:4.0:1496184927.599015:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.599019:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8850 x1568861104085232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.599037:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085232:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.599043:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.599048:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.599051:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.599053:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.599059:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.599061:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.599062:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.599067:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.599070:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.599071:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.599072:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.599073:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.599075:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.599079:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.599080:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.599083:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.599084:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.599087:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.599089:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.599091:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.599092:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.599096:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085232 00000100:00000040:4.0:1496184927.599100:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.599103:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.599114:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.599117:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8850 x1568861104085232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.599123:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085232:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (202us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.599128:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42871 00000100:00000040:4.0:1496184927.599129:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.599133:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.599716:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.599719:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.599721:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.599726:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.599734:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.599741:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.599743:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000100:00100000:4.0:1496184927.599753:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57014 00000100:00000040:4.0:1496184927.599755:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.599757:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.599763:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.599808:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.601187:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bbc00 00000100:00020000:3.0:1496184927.612215:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bbc00 00010000:00020000:4.0:1496184927.623279:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.623324:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.623329:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.623332:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.623337:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.623339:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.623345:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23582us (23674us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.623350:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57014 00000100:00000040:4.0:1496184927.623354:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.623358:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.623364:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.623366:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000020:00000040:4.0:1496184927.623370:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.633804:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085264 00000020:00000040:4.0:1496184927.633807:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.633809:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184927.633823:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42872 00000100:00000040:4.0:1496184927.633825:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.633828:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5050 x1568861104085264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.633835:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085264:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184927.633841:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184927.633846:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184927.633848:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184927.633851:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:4.0:1496184927.633855:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184927.633857:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184927.633858:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184927.633863:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184927.633865:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.633867:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.633868:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184927.633868:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.633873:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.633877:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.633878:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184927.633880:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.633882:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184927.633885:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184927.633887:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184927.633889:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184927.633903:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184927.633907:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085264 00000100:00000040:4.0:1496184927.633911:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.633914:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.633925:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184927.633928:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5050 x1568861104085264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184927.633934:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085264:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (183us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184927.633938:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42872 00000100:00000040:4.0:1496184927.633940:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184927.633943:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.634544:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.634547:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.634549:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.634554:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.634562:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.634568:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.634571:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000100:00100000:4.0:1496184927.634583:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57015 00000100:00000040:4.0:1496184927.634584:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.634587:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.634593:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.634637:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.636176:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9600 00000100:00020000:3.0:1496184927.647222:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9600 00010000:00020000:4.0:1496184927.658323:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184927.658340:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ac50 x1568861104085296 msgsize 520 00000100:00100000:1.0:1496184927.658349:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.658350:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:4.0:1496184927.658381:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.658386:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.658389:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.658394:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.658396:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184927.658398:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085296 00000100:00100000:4.0:1496184927.658402:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23809us (23908us total) trans 0 rc -110/-999 00000020:00000040:2.0:1496184927.658403:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184927.658406:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:4.0:1496184927.658407:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57015 00000100:00000040:4.0:1496184927.658409:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.658413:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.658418:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.658421:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000020:00000040:4.0:1496184927.658425:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184927.658435:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5982 00000100:00000040:2.0:1496184927.658438:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.658443:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ac50 x1568861104085296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184927.658458:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085296:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184927.658470:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184927.658482:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184927.658486:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184927.658490:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:2.0:1496184927.658498:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184927.658501:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184927.658503:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184927.658512:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184927.658516:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.658518:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.658519:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184927.658520:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.658523:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.658528:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.658531:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184927.658534:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.658536:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.658541:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184927.658544:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184927.658546:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.658548:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184927.658556:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085296 00000100:00000040:2.0:1496184927.658563:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.658567:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.658588:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.658590:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ac50 x1568861104085296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184927.658596:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085296:12345-10.37.248.124@o2ib1:8 Request procesed in 139us (256us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184927.658601:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5982 00000100:00000040:2.0:1496184927.658603:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.658609:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184927.659568:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184927.659571:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184927.659575:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184927.659580:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184927.659588:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184927.659601:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.659603:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000100:00100000:4.0:1496184927.659613:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57016 00000100:00000040:4.0:1496184927.659615:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184927.659617:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/0 e 0 to 0 dl 1496184970 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184927.659623:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184927.659668:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184927.670349:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb200 00000100:00020000:3.0:1496184927.681358:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb200 00010000:00020000:4.0:1496184927.692417:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184927.692475:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184927.692480:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184927.692483:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184927.692488:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184927.692491:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184927.692497:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32873us (32974us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184927.692502:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57016 00000100:00000040:4.0:1496184927.692503:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184927.692507:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:715/0 lens 608/448 e 0 to 0 dl 1496184970 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184927.692519:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184927.692522:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000020:00000040:4.0:1496184927.692525:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184927.705541:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184927.705552:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184927.736777:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184927.736787:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184927.768273:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184927.768288:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184927.799331:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184927.810488:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184927.834818:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184927.845861:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184927.857001:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88850 x1568861104085488 msgsize 520 00000100:00100000:1.0:1496184927.857010:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184927.857012:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184927.857065:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085488 00000020:00000040:2.0:1496184927.857070:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184927.857074:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184927.857101:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5983 00000100:00000040:2.0:1496184927.857104:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.857109:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88850 x1568861104085488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184947 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184927.857117:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085488:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184927.857129:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184927.857141:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184927.857146:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184927.857149:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184927 last 1496184927 00000001:00000040:2.0:1496184927.857158:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184927.857161:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184927.857163:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184927.857171:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184927.857175:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.857177:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.857178:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184927.857182:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184927.857185:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.857191:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.857193:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184927.857196:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.857198:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184927.857203:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184927.857206:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184927.857208:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184927.857210:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184927.857218:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085488 00000100:00000040:2.0:1496184927.857226:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.857229:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.857249:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184927.857252:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88850 x1568861104085488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184947 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184927.857258:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085488:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (256us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184927.857263:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5983 00000100:00000040:2.0:1496184927.857265:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184927.857270:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184927.870214:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184927.881213:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184927.905741:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184927.905752:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184927.936406:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184927.936431:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184927.967755:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184927.967780:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184927.998387:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184928.009549:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:4.0:1496184928.032483:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.032490:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.032491:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.034299:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184928.045360:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:1.0:1496184928.056436:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89c50 x1568861104085680 msgsize 520 00000100:00100000:1.0:1496184928.056445:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184928.056447:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184928.056478:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085680 00000020:00000040:3.0:1496184928.056482:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184928.056485:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184928.056510:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5984 00000100:00000040:3.0:1496184928.056512:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184928.056517:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89c50 x1568861104085680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184948 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184928.056526:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104085680:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184928.056538:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184928.056551:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184928.056556:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184928.056559:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184928 last 1496184927 00000001:00000040:3.0:1496184928.056568:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184928.056571:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184928.056573:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184928.056581:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184928.056586:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.056588:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.056589:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184928.056590:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.056593:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.056598:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.056601:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184928.056604:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.056606:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.056614:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184928.056617:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184928.056619:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.056621:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184928.056629:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085680 00000100:00000040:3.0:1496184928.056635:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.056639:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.056658:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.056661:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89c50 x1568861104085680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184948 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184928.056667:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085680:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184928.056672:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5984 00000100:00000040:3.0:1496184928.056674:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.056679:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184928.057605:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.057612:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.057613:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.059377:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184928.070390:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:4.0:1496184928.092949:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.092956:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.092957:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.094557:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.105606:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.127048:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184928.138085:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:4.0:1496184928.159962:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1c50 x1568861104085776 msgsize 520 00000100:00100000:4.0:1496184928.159969:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.159971:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.162378:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184928.173406:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:4.0:1496184928.195925:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.195939:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.195940:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.197953:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.209000:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:4.0:1496184928.230499:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1850 x1568861104085840 msgsize 520 00000100:00100000:4.0:1496184928.230506:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.230507:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.233002:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184928.244053:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:2.0:1496184928.255154:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89850 x1568861104085888 msgsize 520 00000100:00100000:2.0:1496184928.255163:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184928.255164:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184928.255185:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104085888 00000020:00000040:3.0:1496184928.255188:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184928.255192:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184928.255220:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5985 00000100:00000040:3.0:1496184928.255222:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.255228:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89850 x1568861104085888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184948 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184928.255236:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085888:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184928.255249:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184928.255261:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184928.255266:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184928.255270:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184928 last 1496184927 00000001:00000040:3.0:1496184928.255278:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184928.255281:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184928.255283:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184928.255291:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184928.255295:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.255297:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.255298:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184928.255300:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.255303:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.255308:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.255310:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184928.255313:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.255315:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.255321:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184928.255323:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184928.255326:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.255327:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184928.255336:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104085888 00000100:00000040:3.0:1496184928.255344:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.255347:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.255367:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.255370:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89850 x1568861104085888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184948 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184928.255376:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104085888:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184928.255381:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5985 00000100:00000040:3.0:1496184928.255383:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.255388:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184928.256291:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076c50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.256298:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.256300:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.258171:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184928.269191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:4.0:1496184928.291801:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0450 x1568861104085920 msgsize 520 00000100:00100000:4.0:1496184928.291808:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.291809:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184928.292491:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.292505:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.292511:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.294163:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184928.305191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:4.0:1496184928.327027:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1450 x1568861104085952 msgsize 520 00000100:00100000:4.0:1496184928.327034:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.327035:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184928.327747:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.327754:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.327756:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.329490:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184928.340541:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:4.0:1496184928.362908:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.362921:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.362923:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.364828:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184928.375872:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184928.398608:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184928.409638:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:4.0:1496184928.431872:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.431879:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.431884:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.433422:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184928.444469:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:1.0:1496184928.455574:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88050 x1568861104086080 msgsize 520 00000100:00100000:1.0:1496184928.455583:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184928.455585:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184928.455605:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086080 00000020:00000040:3.0:1496184928.455608:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184928.455612:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184928.455637:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5986 00000100:00000040:3.0:1496184928.455639:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.455644:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88050 x1568861104086080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184948 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184928.455660:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086080:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184928.455673:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184928.455685:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184928.455690:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184928.455693:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184928 last 1496184927 00000001:00000040:3.0:1496184928.455701:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184928.455704:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184928.455707:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184928.455715:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184928.455719:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.455721:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.455722:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184928.455723:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.455727:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.455732:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.455734:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184928.455738:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.455740:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.455745:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184928.455748:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184928.455750:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.455752:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184928.455760:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086080 00000100:00000040:3.0:1496184928.455767:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.455771:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.455790:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.455793:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88050 x1568861104086080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184948 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184928.455799:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086080:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (224us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184928.455804:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5986 00000100:00000040:3.0:1496184928.455806:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.455811:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184928.456698:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.456705:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.456706:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.458280:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.469293:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.494570:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184928.505598:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:4.0:1496184928.527390:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104086144 msgsize 520 00000100:00100000:4.0:1496184928.527397:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.527399:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184928.528163:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.528170:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.528171:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.529802:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184928.540842:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000040:4.0:1496184928.562620:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104086176 msgsize 520 00000100:00100000:4.0:1496184928.562627:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.562628:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:4.0:1496184928.563336:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.563343:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.563344:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.565111:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184928.576154:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:4.0:1496184928.598686:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.598695:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.598696:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.600532:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184928.611564:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:4.0:1496184928.633723:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074050 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184928.633730:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184928.633731:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184928.635391:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184928.646438:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:1.0:1496184928.657539:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88450 x1568861104086272 msgsize 520 00000100:00100000:1.0:1496184928.657548:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184928.657550:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184928.657570:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086272 00000020:00000040:3.0:1496184928.657574:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184928.657577:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184928.657602:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5987 00000100:00000040:3.0:1496184928.657604:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.657610:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88450 x1568861104086272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184948 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184928.657618:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104086272:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184928.657631:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184928.657646:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184928.657650:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184928.657653:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184928 last 1496184927 00000001:00000040:3.0:1496184928.657661:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184928.657665:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184928.657667:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184928.657675:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184928.657679:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.657681:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.657683:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184928.657684:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.657687:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.657692:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.657694:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184928.657698:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.657700:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.657705:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184928.657708:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184928.657710:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.657711:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184928.657720:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086272 00000100:00000040:3.0:1496184928.657727:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.657730:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.657750:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.657752:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88450 x1568861104086272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184948 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184928.657759:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086272:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (219us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184928.657764:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5987 00000100:00000040:3.0:1496184928.657766:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.657772:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184928.670190:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184928.681191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184928.705588:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184928.705612:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184928.736389:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.736416:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.767730:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.767741:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.798212:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.809376:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184928.833199:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.844272:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:1.0:1496184928.855390:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88c50 x1568861104086464 msgsize 520 00000100:00100000:1.0:1496184928.855399:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184928.855400:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184928.855420:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086464 00000020:00000040:3.0:1496184928.855424:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184928.855427:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184928.855453:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5988 00000100:00000040:3.0:1496184928.855455:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184928.855460:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88c50 x1568861104086464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184948 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184928.855469:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104086464:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184928.855481:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184928.855494:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184928.855498:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184928.855502:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184928 last 1496184927 00000001:00000040:3.0:1496184928.855510:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184928.855516:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184928.855518:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184928.855526:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184928.855530:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.855532:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.855534:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184928.855535:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184928.855538:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.855543:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.855546:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184928.855549:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.855551:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184928.855556:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184928.855559:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184928.855561:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184928.855563:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184928.855570:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086464 00000100:00000040:3.0:1496184928.855578:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.855581:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.855600:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184928.855603:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88c50 x1568861104086464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184948 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184928.855609:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086464:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (218us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184928.855614:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5988 00000100:00000040:3.0:1496184928.855616:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184928.855622:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184928.858186:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.869206:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184928.893932:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184928.904983:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184928.929343:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184928.940388:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184928.964924:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184928.975973:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:0.0:1496184929.000324:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:0.0:1496184929.011362:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:1.0:1496184929.027301:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:1.0:1496184929.038310:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.049986:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086656 00000020:00000040:4.0:1496184929.049989:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.049990:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.050005:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42908 00000100:00000040:4.0:1496184929.050006:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.050009:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc850 x1568861104086656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.050017:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086656:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.050022:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.050027:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.050030:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.050033:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.050037:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.050038:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.050040:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.050044:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.050047:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.050048:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.050049:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.050050:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.050052:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.050056:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.050059:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.050062:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.050064:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.050066:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.050068:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.050070:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.050071:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.050075:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086656 00000100:00000040:4.0:1496184929.050079:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.050082:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.050093:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.050096:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc850 x1568861104086656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.050102:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086656:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (165us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.050106:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42908 00000100:00000040:4.0:1496184929.050108:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.050111:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:1.0:1496184929.052195:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:1.0:1496184929.063234:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.074833:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086688 00000020:00000040:4.0:1496184929.074836:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.074838:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.074852:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42909 00000100:00000040:4.0:1496184929.074854:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.074857:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdcc50 x1568861104086688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.074864:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086688:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.074869:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.074875:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.074880:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.074882:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.074886:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.074888:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.074905:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.074910:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.074913:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.074914:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.074915:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.074916:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.074918:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.074922:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.074923:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.074926:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.074928:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.074938:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.074940:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.074942:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.074943:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.074946:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086688 00000100:00000040:4.0:1496184929.074951:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.074953:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.074965:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.074968:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdcc50 x1568861104086688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.074974:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086688:12345-10.37.248.124@o2ib1:8 Request procesed in 110us (186us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.074979:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42909 00000100:00000040:4.0:1496184929.074980:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.074984:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:1.0:1496184929.077274:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:1.0:1496184929.088266:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.099857:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086720 00000020:00000040:4.0:1496184929.099860:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.099862:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.099876:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42910 00000100:00000040:4.0:1496184929.099878:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.099881:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd050 x1568861104086720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.099888:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086720:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.099901:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.099906:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.099909:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.099911:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.099915:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.099917:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.099918:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.099923:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.099926:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.099927:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.099928:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.099929:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.099931:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.099935:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.099936:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.099941:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.099943:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.099945:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.099948:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.099949:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.099950:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.099954:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086720 00000100:00000040:4.0:1496184929.099958:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.099961:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.099972:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.099975:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd050 x1568861104086720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.099981:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086720:12345-10.37.248.124@o2ib1:8 Request procesed in 93us (167us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.099985:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42910 00000100:00000040:4.0:1496184929.099987:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.099990:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.102282:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.113287:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.134751:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086752 00000020:00000040:4.0:1496184929.134754:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.134756:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.134770:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42911 00000100:00000040:4.0:1496184929.134772:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.134774:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd450 x1568861104086752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.134782:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086752:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.134787:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.134793:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.134796:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.134800:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.134804:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.134806:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.134807:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.134812:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.134814:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.134816:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.134816:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.134817:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.134819:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.134823:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.134825:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.134827:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.134829:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.134831:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.134834:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.134835:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.134836:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.134840:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086752 00000100:00000040:4.0:1496184929.134844:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.134847:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.134858:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.134861:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd450 x1568861104086752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.134867:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086752:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (162us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.134874:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42911 00000100:00000040:4.0:1496184929.134876:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.134879:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.137303:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.148327:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.170463:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086784 00000020:00000040:4.0:1496184929.170466:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.170468:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.170482:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42912 00000100:00000040:4.0:1496184929.170484:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.170487:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd850 x1568861104086784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.170494:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086784:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.170500:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.170505:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.170508:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.170511:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.170514:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.170516:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.170517:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.170522:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.170525:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.170526:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.170527:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.170528:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.170530:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.170534:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.170535:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.170540:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.170541:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.170544:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.170546:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.170548:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.170549:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.170553:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086784 00000100:00000040:4.0:1496184929.170557:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.170559:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.170571:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.170574:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd850 x1568861104086784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.170580:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086784:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (164us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.170584:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42912 00000100:00000040:4.0:1496184929.170586:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.170589:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.172724:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.183756:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.205871:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086816 00000020:00000040:4.0:1496184929.205874:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.205876:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.205904:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42913 00000100:00000040:4.0:1496184929.205906:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.205909:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104086816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.205916:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086816:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.205922:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.205927:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.205930:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.205934:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.205938:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.205940:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.205941:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.205946:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.205949:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.205950:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.205951:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.205952:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.205954:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.205958:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.205959:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.205962:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.205964:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.205966:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.205969:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.205970:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.205972:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.205975:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086816 00000100:00000040:4.0:1496184929.205979:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.205982:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.205993:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.205996:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104086816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.206002:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086816:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (179us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.206009:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42913 00000100:00000040:4.0:1496184929.206011:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.206014:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.208377:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.219407:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.241175:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086848 00000020:00000040:4.0:1496184929.241178:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.241179:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.241194:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42914 00000100:00000040:4.0:1496184929.241196:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.241198:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104086848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.241206:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086848:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.241212:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.241217:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.241219:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.241222:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.241226:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.241228:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.241229:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.241234:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.241236:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.241238:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.241239:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.241239:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.241241:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.241245:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.241247:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.241251:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.241253:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.241255:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.241258:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.241260:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.241261:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.241264:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086848 00000100:00000040:4.0:1496184929.241268:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.241271:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.241283:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.241285:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104086848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.241292:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086848:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (164us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.241296:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42914 00000100:00000040:4.0:1496184929.241298:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.241301:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.243478:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.254506:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00100000:4.0:1496184929.269379:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104086880 00000020:00000040:4.0:1496184929.269382:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.269383:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184929.269398:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42915 00000100:00000040:4.0:1496184929.269400:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.269403:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104086880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.269410:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086880:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184929.269416:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184929.269421:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184929.269424:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184929.269431:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:4.0:1496184929.269435:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184929.269437:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184929.269438:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184929.269443:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184929.269445:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.269447:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.269448:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184929.269448:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.269450:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.269454:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.269456:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184929.269458:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.269460:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184929.269463:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184929.269465:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184929.269467:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184929.269468:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184929.269471:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104086880 00000100:00000040:4.0:1496184929.269476:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.269478:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.269490:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184929.269493:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104086880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184929.269499:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104086880:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (168us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184929.269506:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42915 00000100:00000040:4.0:1496184929.269507:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184929.269511:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.271755:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.282737:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184929.307291:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184929.307302:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184929.337959:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184929.337970:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:2.0:1496184929.340907:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184929.340931:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184929.340951:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.340969:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f769e00] to pc [ptlrpcd_00_03:3] req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.340979:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184929.340982:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184929.340986:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.340991:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76ad00] to pc [ptlrpcd_00_00:0] req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.341001:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1496184929.341002:0:13691:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000020:00000040:2.0:1496184929.341003:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184929.341006:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.341011:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76a700] to pc [ptlrpcd_00_01:1] req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341013:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.341018:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 5000 (4297368369) 00000100:00100000:3.0:1496184929.341024:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:d86c02e9-1939-02b8-4331-d3612fcf0106:13691:1568861159710096:10.37.248.196@o2ib1:400 00000100:00000040:2.0:1496184929.341026:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184929.341033:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710128:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184929.341035:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341044:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.341044:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184929.341045:0:13689:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710112:10.37.248.197@o2ib1:400 00000100:00000040:3.0:1496184929.341051:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:2.0:1496184929.341051:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:0.0:1496184929.341057:0:13689:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184929.341063:0:13689:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184929.341101:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341109:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341115:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341427:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184929.341443:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184929.341459:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184929.341467:0:13690:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:2.0:1496184929.341470:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184929.341476:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184929.341480:0:13689:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000040:1.0:1496184929.341483:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00100000:2.0:1496184929.341486:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710128:10.37.248.197@o2ib1:400 00000100:00000040:2.0:1496184929.341489:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76a700 x1568861159710128/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184929.341489:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Interpret:RN/0/0 rc 0/0 00000020:00000040:2.0:1496184929.341495:0:13690:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00100000:1.0:1496184929.341497:0:13689:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710112:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184929.341498:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184929.341500:0:13689:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76ad00 x1568861159710112/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:1.0:1496184929.341505:0:13689:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:0.0:1496184929.341506:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:0.0:1496184929.341518:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:d86c02e9-1939-02b8-4331-d3612fcf0106:13691:1568861159710096:10.37.248.196@o2ib1:400 00000100:00000040:0.0:1496184929.341521:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f769e00 x1568861159710096/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184936 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:0.0:1496184929.341527:0:13691:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000100:00020000:3.0:1496184929.368550:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184929.368575:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00100000:4.0:1496184929.397617:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.397620:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.397622:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.397627:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.397635:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.397642:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.397644:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000100:00100000:4.0:1496184929.397654:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57069 00000100:00000040:4.0:1496184929.397656:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.397658:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.397665:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.397710:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.399410:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00020000:3.0:1496184929.410551:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00010000:00020000:4.0:1496184929.421692:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.421737:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.421742:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.421745:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.421750:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.421752:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.421764:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24099us (24202us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.421769:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57069 00000100:00000040:4.0:1496184929.421771:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.421774:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.421780:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.421783:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000020:00000040:4.0:1496184929.421786:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.432772:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.432775:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.432777:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.432783:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.432791:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.432798:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.432800:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:4.0:1496184929.432810:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57070 00000100:00000040:4.0:1496184929.432812:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.432814:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.432821:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.432864:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.434265:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00020000:3.0:1496184929.445312:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00010000:00020000:4.0:1496184929.456411:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184929.456429:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285050 x1568861104087072 msgsize 520 00000100:00100000:1.0:1496184929.456438:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184929.456439:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:4.0:1496184929.456456:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184929.456460:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087072 00000020:02000400:4.0:1496184929.456463:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184929.456463:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00010000:00000080:4.0:1496184929.456466:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184929.456467:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000040:4.0:1496184929.456471:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.456474:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.456480:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23659us (23751us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.456485:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57070 00000100:00000040:4.0:1496184929.456486:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.456491:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184929.456491:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5989 00000100:00000040:3.0:1496184929.456494:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:4.0:1496184929.456497:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.456499:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00000040:3.0:1496184929.456499:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285050 x1568861104087072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000040:4.0:1496184929.456503:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184929.456508:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087072:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184929.456520:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184929.456532:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184929.456537:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184929.456540:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:3.0:1496184929.456553:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184929.456556:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184929.456558:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184929.456566:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184929.456570:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.456573:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.456574:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184929.456575:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.456578:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.456584:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.456586:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184929.456589:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.456591:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.456596:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184929.456599:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184929.456602:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.456603:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184929.456611:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087072 00000100:00000040:3.0:1496184929.456618:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.456622:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.456640:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.456643:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285050 x1568861104087072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184929.456649:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087072:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184929.456654:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5989 00000100:00000040:3.0:1496184929.456656:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.456662:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.457619:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.457622:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.457624:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.457630:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.457638:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.457644:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.457647:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:4.0:1496184929.457657:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57071 00000100:00000040:4.0:1496184929.457659:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.457661:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.457673:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.457717:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.470203:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bbc00 00000100:00020000:3.0:1496184929.481207:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bbc00 00010000:00020000:4.0:1496184929.492265:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184929.492284:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89050 x1568861104087104 msgsize 520 00000100:00100000:1.0:1496184929.492292:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184929.492294:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184929.492308:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087104 00000020:00000040:3.0:1496184929.492310:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184929.492312:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:4.0:1496184929.492322:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184929.492328:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000100:00100000:3.0:1496184929.492329:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5990 00010000:00000080:4.0:1496184929.492331:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184929.492332:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184929.492335:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89050 x1568861104087104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00010000:00000040:4.0:1496184929.492336:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.492339:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184929.492343:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104087104:12345-10.37.248.124@o2ib1:8 00000100:00100000:4.0:1496184929.492346:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34672us (34772us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.492351:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57071 00000020:00000040:3.0:1496184929.492352:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000040:4.0:1496184929.492353:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:4.0:1496184929.492357:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184929.492359:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00002000:00100000:4.0:1496184929.492363:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00010000:02000400:3.0:1496184929.492363:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00002000:00010000:4.0:1496184929.492365:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00010000:00080000:3.0:1496184929.492366:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000020:00000040:4.0:1496184929.492369:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000001:00000040:3.0:1496184929.492371:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184929.492374:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184929.492375:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184929.492381:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184929.492384:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.492386:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.492387:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184929.492388:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.492390:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.492395:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.492397:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184929.492400:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.492405:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.492408:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184929.492411:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184929.492413:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.492414:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184929.492419:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087104 00000100:00000040:3.0:1496184929.492425:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.492428:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.492443:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.492445:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89050 x1568861104087104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184929.492451:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087104:12345-10.37.248.124@o2ib1:8 Request procesed in 108us (166us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184929.492456:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5990 00000100:00000040:3.0:1496184929.492458:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.492462:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.493486:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.493489:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.493491:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.493496:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.493504:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.493517:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.493520:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000100:00100000:4.0:1496184929.493530:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57072 00000100:00000040:4.0:1496184929.493531:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.493534:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.493540:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.493586:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.495244:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9200 00000100:00020000:3.0:1496184929.506257:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9200 00010000:00020000:4.0:1496184929.517322:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.517380:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.517385:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.517388:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.517393:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.517396:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.517402:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23862us (23961us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.517407:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57072 00000100:00000040:4.0:1496184929.517409:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.517412:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.517418:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.517423:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000020:00000040:4.0:1496184929.517426:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.528763:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.528766:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.528768:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.528773:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.528781:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.528788:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.528790:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000100:00100000:4.0:1496184929.528800:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57073 00000100:00000040:4.0:1496184929.528802:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.528804:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.528810:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.528855:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.530472:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb200 00000100:00020000:3.0:1496184929.541501:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb200 00010000:00020000:4.0:1496184929.552554:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.552612:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.552617:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.552620:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.552625:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.552628:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.552634:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23823us (23916us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.552643:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57073 00000100:00000040:4.0:1496184929.552644:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.552648:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.552654:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.552656:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000020:00000040:4.0:1496184929.552660:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.564202:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.564205:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.564206:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.564212:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.564220:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.564226:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.564229:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000100:00100000:4.0:1496184929.564239:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57074 00000100:00000040:4.0:1496184929.564240:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.564243:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.564249:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.564291:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.565780:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105baa00 00000100:00020000:3.0:1496184929.576836:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105baa00 00010000:00020000:4.0:1496184929.587884:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.587954:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.587958:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.587962:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.587969:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.587972:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.587978:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23728us (23829us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.587982:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57074 00000100:00000040:4.0:1496184929.587984:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.587988:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.587993:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.587996:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000020:00000040:4.0:1496184929.587999:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.599193:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.599196:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.599198:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.599204:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.599211:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.599218:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.599220:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:4.0:1496184929.599230:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57075 00000100:00000040:4.0:1496184929.599232:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.599234:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.599240:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.599284:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.601053:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb400 00000100:00020000:3.0:1496184929.612088:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb400 00010000:00020000:4.0:1496184929.623152:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.623210:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.623215:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.623218:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.623223:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.623225:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.623231:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23991us (24084us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.623236:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57075 00000100:00000040:4.0:1496184929.623238:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.623241:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.623247:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.623250:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:4.0:1496184929.623253:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.634251:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.634254:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.634256:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.634261:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.634269:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.634276:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.634278:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:4.0:1496184929.634288:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57076 00000100:00000040:4.0:1496184929.634291:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.634294:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.634300:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.634346:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.635782:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9c00 00000100:00020000:3.0:1496184929.646833:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9c00 00010000:00020000:4.0:1496184929.657918:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184929.657948:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286050 x1568861104087280 msgsize 520 00000100:00100000:1.0:1496184929.657957:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184929.657958:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:4.0:1496184929.657981:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.657985:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.657989:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.657993:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.657996:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.658002:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23702us (23795us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184929.658006:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087280 00000100:00100000:4.0:1496184929.658007:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57076 00000100:00000040:4.0:1496184929.658009:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.658011:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00100000:4.0:1496184929.658012:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184929.658014:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00100000:4.0:1496184929.658018:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.658021:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000020:00000040:4.0:1496184929.658025:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184929.658043:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5991 00000100:00000040:3.0:1496184929.658047:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.658053:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286050 x1568861104087280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184929.658061:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087280:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184929.658073:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184929.658085:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184929.658090:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184929.658093:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:3.0:1496184929.658102:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184929.658105:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184929.658108:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184929.658116:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184929.658120:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.658122:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.658123:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184929.658124:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.658127:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.658133:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.658135:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184929.658138:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.658141:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.658145:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184929.658148:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184929.658151:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.658152:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184929.658160:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087280 00000100:00000040:3.0:1496184929.658168:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.658172:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.658192:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.658195:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286050 x1568861104087280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184929.658201:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087280:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184929.658206:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5991 00000100:00000040:3.0:1496184929.658208:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.658213:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184929.659126:0:13742:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184929.659129:0:13742:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184929.659131:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184929.659139:0:13742:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184929.659147:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184929.659153:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.659156:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:4.0:1496184929.659165:0:13742:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57077 00000100:00000040:4.0:1496184929.659167:0:13742:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184929.659169:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/0 e 0 to 0 dl 1496184972 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184929.659176:0:13742:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184929.659218:0:13742:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184929.670312:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105baa00 00000100:00020000:3.0:1496184929.681314:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105baa00 00010000:00020000:4.0:1496184929.692370:0:13742:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184929.692415:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184929.692420:0:13742:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184929.692423:0:13742:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184929.692428:0:13742:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184929.692431:0:13742:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184929.692437:0:13742:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_002:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33261us (33356us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184929.692442:0:13742:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57077 00000100:00000040:4.0:1496184929.692443:0:13742:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184929.692447:0:13742:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_002: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:717/0 lens 608/448 e 0 to 0 dl 1496184972 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184929.692459:0:13742:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184929.692462:0:13742:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:4.0:1496184929.692465:0:13742:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.705523:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184929.705548:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184929.736090:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184929.736114:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184929.767134:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184929.767160:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184929.798052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184929.809206:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184929.833153:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184929.844207:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184929.855328:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285450 x1568861104087472 msgsize 520 00000100:00100000:1.0:1496184929.855337:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184929.855339:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184929.855359:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087472 00000020:00000040:3.0:1496184929.855363:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184929.855366:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184929.855391:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5992 00000100:00000040:3.0:1496184929.855393:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.855398:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285450 x1568861104087472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184949 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184929.855407:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087472:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184929.855422:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184929.855434:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184929.855439:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184929.855442:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184929 last 1496184929 00000001:00000040:3.0:1496184929.855459:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184929.855462:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184929.855465:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184929.855473:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184929.855477:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.855480:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.855481:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184929.855482:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184929.855485:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.855490:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.855493:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184929.855496:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.855498:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184929.855503:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184929.855515:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184929.855517:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184929.855519:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184929.855527:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087472 00000100:00000040:3.0:1496184929.855534:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.855538:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.855557:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184929.855559:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285450 x1568861104087472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184949 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184929.855566:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087472:12345-10.37.248.124@o2ib1:8 Request procesed in 160us (237us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184929.855572:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5992 00000100:00000040:3.0:1496184929.855574:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184929.855579:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184929.858229:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184929.869238:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184929.894193:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184929.905226:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184929.930029:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184929.941060:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184929.965609:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184929.976691:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:0.0:1496184930.001357:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:0.0:1496184930.012388:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:0.0:1496184930.028698:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:0.0:1496184930.039702:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00000040:3.0:1496184930.051026:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286450 x1568861104087664 msgsize 520 00000100:00100000:3.0:1496184930.051035:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184930.051036:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184930.051096:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087664 00000020:00000040:0.0:1496184930.051100:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184930.051104:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184930.051134:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5993 00000100:00000040:0.0:1496184930.051136:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184930.051142:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286450 x1568861104087664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184930.051150:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087664:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184930.051163:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184930.051175:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184930.051180:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184930.051184:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:0.0:1496184930.051192:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184930.051195:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184930.051198:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184930.051206:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184930.051211:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184930.051215:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184930.051217:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184930.051218:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184930.051221:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.051226:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184930.051229:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184930.051232:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.051234:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184930.051239:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184930.051242:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184930.051245:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184930.051246:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184930.051255:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087664 00000100:00000040:0.0:1496184930.051262:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.051266:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.051285:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.051288:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286450 x1568861104087664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184930.051294:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087664:12345-10.37.248.124@o2ib1:8 Request procesed in 144us (267us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184930.051299:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5993 00000100:00000040:0.0:1496184930.051301:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184930.051306:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:0.0:1496184930.053912:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:0.0:1496184930.064942:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:0.0:1496184930.078921:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:0.0:1496184930.089938:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:3.0:1496184930.102803:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285850 x1568861104087728 msgsize 520 00000100:00100000:3.0:1496184930.102812:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184930.102813:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184930.102832:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104087728 00000020:00000040:0.0:1496184930.102836:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184930.102842:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184930.102875:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5994 00000100:00000040:0.0:1496184930.102877:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184930.102882:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285850 x1568861104087728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184930.102897:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087728:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184930.102909:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184930.102930:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184930.102934:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184930.102938:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:0.0:1496184930.102945:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184930.102949:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184930.102951:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184930.102959:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184930.102963:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184930.102965:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184930.102966:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184930.102967:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184930.102970:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.102984:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184930.102987:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184930.102990:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.102992:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184930.102997:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184930.103000:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184930.103002:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184930.103005:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184930.103012:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104087728 00000100:00000040:0.0:1496184930.103019:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.103023:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.103042:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184930.103045:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285850 x1568861104087728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184930.103051:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104087728:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (247us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184930.103056:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5994 00000100:00000040:0.0:1496184930.103058:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184930.103064:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.105183:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184930.116187:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184930.140343:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184930.151367:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184930.176147:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184930.187163:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184930.211513:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.222546:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.246363:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.257400:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.272074:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184930.283084:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184930.307466:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184930.307477:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184930.338566:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184930.338576:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184930.369025:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.369052:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.399921:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.411075:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.434989:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.446063:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:1.0:1496184930.457181:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285c50 x1568861104088080 msgsize 520 00000100:00100000:1.0:1496184930.457190:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184930.457191:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184930.457240:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088080 00000020:00000040:2.0:1496184930.457245:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184930.457249:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184930.457277:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5995 00000100:00000040:2.0:1496184930.457279:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.457284:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285c50 x1568861104088080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184930.457293:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088080:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184930.457305:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184930.457317:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184930.457322:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184930.457325:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:2.0:1496184930.457333:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184930.457337:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184930.457339:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184930.457347:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184930.457351:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.457353:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.457355:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184930.457356:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.457359:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.457367:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.457369:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184930.457372:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.457375:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.457380:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184930.457383:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184930.457385:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.457387:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184930.457395:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088080 00000100:00000040:2.0:1496184930.457402:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.457405:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.457425:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.457427:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285c50 x1568861104088080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184930.457433:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088080:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (252us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184930.457438:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5995 00000100:00000040:2.0:1496184930.457441:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.457446:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.470166:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.481158:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.505469:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.505480:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.536651:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.536676:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.567304:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.567329:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.597469:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.608617:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.632661:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.643721:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:1.0:1496184930.654844:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287850 x1568861104088272 msgsize 520 00000100:00100000:1.0:1496184930.654852:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184930.654854:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184930.654878:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088272 00000020:00000040:2.0:1496184930.654883:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184930.654886:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184930.654920:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5996 00000100:00000040:2.0:1496184930.654923:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.654928:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287850 x1568861104088272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184930.654937:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088272:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184930.654950:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184930.654964:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184930.654969:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184930.654972:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:2.0:1496184930.654980:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184930.654983:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184930.654986:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184930.654994:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184930.654998:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.655000:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.655001:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184930.655002:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.655005:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.655011:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.655014:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184930.655017:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.655019:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.655024:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184930.655027:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184930.655029:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.655031:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184930.655039:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088272 00000100:00000040:2.0:1496184930.655046:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.655050:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.655069:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.655072:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287850 x1568861104088272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184930.655078:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088272:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (234us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184930.655084:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5996 00000100:00000040:2.0:1496184930.655086:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.655092:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.657779:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.668782:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.695293:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.706324:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.730555:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.741613:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.765694:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.776741:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184930.801191:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.812232:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184930.835975:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.847027:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:1.0:1496184930.858131:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286c50 x1568861104088464 msgsize 520 00000100:00100000:1.0:1496184930.858140:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184930.858142:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184930.858190:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088464 00000020:00000040:2.0:1496184930.858195:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184930.858198:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184930.858226:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5997 00000100:00000040:2.0:1496184930.858229:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.858234:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286c50 x1568861104088464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184930.858243:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088464:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184930.858255:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184930.858266:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184930.858271:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184930.858275:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:2.0:1496184930.858283:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184930.858286:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184930.858289:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184930.858297:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184930.858301:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.858303:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.858304:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184930.858305:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.858308:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.858314:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.858317:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184930.858322:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.858324:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.858329:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184930.858332:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184930.858335:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.858336:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184930.858344:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088464 00000100:00000040:2.0:1496184930.858352:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.858355:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.858374:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.858376:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286c50 x1568861104088464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184930.858383:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088464:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (251us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184930.858387:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5997 00000100:00000040:2.0:1496184930.858389:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.858395:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.870915:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.881918:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184930.906217:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184930.906241:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:3.0:1496184930.925940:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee5c50 x1568861152397488 msgsize 224 00000100:00100000:3.0:1496184930.925948:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:3.0:1496184930.925949:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184930.925968:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397488 00000020:00000040:1.0:1496184930.925972:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:1.0:1496184930.925975:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:1.0:1496184930.925986:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184930.926001:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 952 00000100:00000040:1.0:1496184930.926003:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:1.0:1496184930.926007:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee5c50 x1568861152397488/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:681/0 lens 224/0 e 0 to 0 dl 1496184936 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184930.926016:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397488:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184930.926036:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:1.0:1496184930.926042:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397488 00000100:00000040:1.0:1496184930.926049:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184930.926053:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184930.926071:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184930.926073:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee5c50 x1568861152397488/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:681/0 lens 224/336 e 0 to 0 dl 1496184936 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184930.926079:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397488:12345-10.37.248.197@o2ib1:13 Request procesed in 64us (139us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184930.926084:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 952 00000100:00000040:1.0:1496184930.926086:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:1.0:1496184930.926090:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00000100:00020000:3.0:1496184930.937571:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.937597:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184930.966568:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287450 x1568861104088560 msgsize 520 00000100:00100000:1.0:1496184930.966575:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184930.966577:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184930.966588:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088560 00000020:00000040:2.0:1496184930.966592:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184930.966594:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184930.966609:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5998 00000100:00000040:2.0:1496184930.966611:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.966614:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287450 x1568861104088560/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184930.966622:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088560:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184930.966630:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184930.966638:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184930.966641:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184930.966647:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:2.0:1496184930.966653:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184930.966656:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184930.966657:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184930.966664:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184930.966668:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.966669:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.966670:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184930.966671:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.966674:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.966678:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.966681:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184930.966683:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.966686:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.966690:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184930.966693:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184930.966695:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.966697:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184930.966703:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088560 00000100:00000040:2.0:1496184930.966707:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.966710:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.966723:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.966726:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287450 x1568861104088560/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184930.966732:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088560:12345-10.37.248.124@o2ib1:8 Request procesed in 110us (162us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184930.966739:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5998 00000100:00000040:2.0:1496184930.966741:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.966744:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.969025:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184930.969050:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:1.0:1496184930.997440:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee5050 x1568861104088592 msgsize 520 00000100:00100000:1.0:1496184930.997455:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184930.997456:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184930.997466:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088592 00000020:00000040:2.0:1496184930.997469:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184930.997470:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184930.997482:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 5999 00000100:00000040:2.0:1496184930.997483:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.997486:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee5050 x1568861104088592/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184950 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184930.997492:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088592:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184930.997498:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184930.997503:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184930.997512:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184930.997515:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184930 last 1496184929 00000001:00000040:2.0:1496184930.997518:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184930.997520:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184930.997521:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184930.997526:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184930.997529:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.997530:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.997531:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184930.997532:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184930.997534:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.997538:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.997539:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184930.997543:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.997545:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184930.997547:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184930.997550:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184930.997551:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184930.997553:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184930.997556:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088592 00000100:00000040:2.0:1496184930.997560:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.997563:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.997573:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184930.997576:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee5050 x1568861104088592/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184950 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184930.997581:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088592:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (140us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184930.997586:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 5999 00000100:00000040:2.0:1496184930.997587:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184930.997590:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184930.999937:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184931.011069:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184931.035454:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.046496:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184931.057605:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286850 x1568861104088656 msgsize 520 00000100:00100000:1.0:1496184931.057614:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.057616:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.057636:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088656 00000020:00000040:3.0:1496184931.057639:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.057643:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184931.057668:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6000 00000100:00000040:3.0:1496184931.057670:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.057676:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286850 x1568861104088656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.057685:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104088656:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.057697:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.057709:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.057716:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.057719:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.057727:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.057730:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.057733:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.057741:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.057745:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.057747:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.057748:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.057749:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.057752:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.057758:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.057760:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.057763:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.057765:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.057771:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.057774:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.057776:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.057778:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.057785:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088656 00000100:00000040:3.0:1496184931.057793:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.057796:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.057815:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.057818:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286850 x1568861104088656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.057824:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088656:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (218us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.057830:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6000 00000100:00000040:3.0:1496184931.057832:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.057838:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.070344:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.081348:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184931.103280:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287050 x1568861104088688 msgsize 520 00000100:00100000:1.0:1496184931.103289:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.103292:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.103332:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088688 00000020:00000040:3.0:1496184931.103336:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.103339:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184931.103360:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6001 00000100:00000040:3.0:1496184931.103362:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.103366:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287050 x1568861104088688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.103374:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088688:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.103382:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.103391:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.103394:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.103397:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.103402:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.103405:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.103406:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.103412:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.103415:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.103416:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.103418:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.103418:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.103421:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.103428:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.103430:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.103432:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.103434:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.103438:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.103441:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.103443:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.103444:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.103449:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088688 00000100:00000040:3.0:1496184931.103454:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.103458:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.103477:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.103480:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287050 x1568861104088688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.103486:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088688:12345-10.37.248.124@o2ib1:8 Request procesed in 113us (205us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.103491:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6001 00000100:00000040:3.0:1496184931.103493:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.103497:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.106071:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.106096:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184931.134450:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4c50 x1568861104088720 msgsize 520 00000100:00100000:1.0:1496184931.134457:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.134458:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.134498:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088720 00000020:00000040:3.0:1496184931.134501:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.134503:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184931.134519:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6002 00000100:00000040:3.0:1496184931.134521:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.134524:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4c50 x1568861104088720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.134532:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088720:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.134540:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.134546:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.134549:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.134551:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.134556:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.134557:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.134559:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.134564:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.134567:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.134568:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.134569:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.134570:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.134572:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.134577:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.134578:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.134581:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.134582:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.134585:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.134587:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.134589:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.134590:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.134595:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088720 00000100:00000040:3.0:1496184931.134599:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.134602:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.134614:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.134616:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4c50 x1568861104088720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.134623:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088720:12345-10.37.248.124@o2ib1:8 Request procesed in 92us (172us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.134628:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6002 00000100:00000040:3.0:1496184931.134630:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.134633:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.137172:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.137197:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.168545:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.168569:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.198791:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.209917:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.234338:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.245374:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184931.256485:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4850 x1568861104088848 msgsize 520 00000100:00100000:1.0:1496184931.256494:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.256495:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.256515:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088848 00000020:00000040:3.0:1496184931.256519:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.256522:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184931.256548:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6003 00000100:00000040:3.0:1496184931.256550:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184931.256556:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4850 x1568861104088848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.256565:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104088848:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.256577:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.256590:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.256594:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.256598:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.256606:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.256609:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.256614:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.256622:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.256626:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.256628:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.256629:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.256630:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.256634:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.256639:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.256641:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.256644:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.256647:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.256652:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.256655:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.256657:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.256658:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.256666:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088848 00000100:00000040:3.0:1496184931.256673:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.256677:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.256695:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.256698:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4850 x1568861104088848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.256704:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088848:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (219us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.256709:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6003 00000100:00000040:3.0:1496184931.256711:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.256717:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.270233:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.281225:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:1.0:1496184931.303104:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3c50 x1568861104088880 msgsize 520 00000100:00100000:1.0:1496184931.303112:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.303114:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.303155:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104088880 00000020:00000040:3.0:1496184931.303158:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.303161:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184931.303181:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6004 00000100:00000040:3.0:1496184931.303183:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.303187:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3c50 x1568861104088880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.303195:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088880:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.303203:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.303211:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.303214:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.303217:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.303222:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.303225:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.303227:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.303232:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.303236:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.303237:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.303239:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.303240:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.303242:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.303246:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.303248:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.303251:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.303253:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.303257:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.303262:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.303264:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.303266:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.303273:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104088880 00000100:00000040:3.0:1496184931.303278:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.303281:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.303297:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.303300:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3c50 x1568861104088880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.303306:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104088880:12345-10.37.248.124@o2ib1:8 Request procesed in 112us (201us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.303310:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6004 00000100:00000040:3.0:1496184931.303312:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.303317:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.305770:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.305794:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.336276:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184931.336301:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184931.366917:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184931.366941:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184931.397505:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184931.408627:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184931.432272:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184931.443309:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184931.454417:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4450 x1568861104089040 msgsize 520 00000100:00100000:1.0:1496184931.454426:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.454428:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.454448:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089040 00000020:00000040:3.0:1496184931.454451:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.454455:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184931.454480:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6005 00000100:00000040:3.0:1496184931.454482:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.454487:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4450 x1568861104089040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.454498:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089040:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.454511:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.454523:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.454527:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.454531:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.454539:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.454542:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.454544:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.454552:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.454556:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.454558:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.454559:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.454561:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.454564:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.454569:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.454571:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.454574:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.454577:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.454582:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.454584:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.454587:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.454588:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.454596:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089040 00000100:00000040:3.0:1496184931.454603:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.454607:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.454626:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.454629:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4450 x1568861104089040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.454635:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089040:12345-10.37.248.124@o2ib1:8 Request procesed in 138us (217us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.454640:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6005 00000100:00000040:3.0:1496184931.454642:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.454648:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.457531:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184931.468515:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184931.495509:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184931.506536:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184931.530723:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.541758:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.565927:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.576960:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.601003:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.612020:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.635852:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.646898:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000040:0.0:1496184931.657970:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3450 x1568861104089232 msgsize 520 00000100:00100000:0.0:1496184931.657979:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184931.657981:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.658001:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089232 00000020:00000040:3.0:1496184931.658004:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.658008:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184931.658032:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6006 00000100:00000040:3.0:1496184931.658035:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184931.658040:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3450 x1568861104089232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.658049:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104089232:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.658062:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.658077:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.658081:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.658085:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.658093:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.658096:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.658098:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.658106:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.658117:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.658119:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.658120:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.658121:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.658125:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.658130:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.658133:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.658136:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.658138:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.658143:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.658147:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.658149:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.658151:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.658158:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089232 00000100:00000040:3.0:1496184931.658166:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.658170:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.658190:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.658193:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3450 x1568861104089232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.658199:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089232:12345-10.37.248.124@o2ib1:8 Request procesed in 150us (228us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.658205:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6006 00000100:00000040:3.0:1496184931.658207:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.658213:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.670203:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.681187:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000040:3.0:1496184931.702964:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3850 x1568861104089264 msgsize 520 00000100:00100000:3.0:1496184931.702972:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184931.702973:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184931.703018:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089264 00000020:00000040:0.0:1496184931.703021:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184931.703024:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184931.703045:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6007 00000100:00000040:0.0:1496184931.703047:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184931.703050:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3850 x1568861104089264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184931.703058:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089264:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184931.703066:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184931.703074:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184931.703077:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184931.703080:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:0.0:1496184931.703086:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184931.703091:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184931.703093:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184931.703099:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184931.703102:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184931.703104:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184931.703105:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184931.703106:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184931.703108:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184931.703113:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184931.703115:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184931.703117:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184931.703120:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184931.703123:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184931.703125:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184931.703127:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184931.703129:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184931.703135:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089264 00000100:00000040:0.0:1496184931.703140:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184931.703144:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184931.703164:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184931.703166:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3850 x1568861104089264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184931.703172:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089264:12345-10.37.248.124@o2ib1:8 Request procesed in 115us (207us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184931.703177:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6007 00000100:00000040:0.0:1496184931.703179:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184931.703183:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.705576:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.705602:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184931.736332:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.736358:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184931.767528:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184931.767552:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184931.797576:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184931.808690:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184931.832592:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184931.843641:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:1.0:1496184931.854741:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2450 x1568861104089424 msgsize 520 00000100:00100000:1.0:1496184931.854751:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184931.854753:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184931.854773:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089424 00000020:00000040:3.0:1496184931.854776:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184931.854780:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184931.854803:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6008 00000100:00000040:3.0:1496184931.854806:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184931.854811:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2450 x1568861104089424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184951 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184931.854820:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104089424:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184931.854832:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184931.854844:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184931.854849:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184931.854855:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184931 last 1496184931 00000001:00000040:3.0:1496184931.854863:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184931.854866:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184931.854869:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184931.854877:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184931.854881:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.854883:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.854884:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184931.854885:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184931.854888:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.854901:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.854904:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184931.854907:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.854909:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184931.854914:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184931.854917:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184931.854920:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184931.854922:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184931.854930:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089424 00000100:00000040:3.0:1496184931.854937:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.854940:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.854959:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184931.854961:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2450 x1568861104089424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184951 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184931.854968:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089424:12345-10.37.248.124@o2ib1:8 Request procesed in 148us (225us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184931.854976:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6008 00000100:00000040:3.0:1496184931.854978:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184931.854983:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184931.857580:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184931.868568:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184931.895717:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184931.906731:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184931.930962:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184931.942021:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184931.966329:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184931.977369:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.001788:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.012823:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:0.0:1496184932.036722:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:0.0:1496184932.047849:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00100000:4.0:1496184932.058913:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089616 00000020:00000040:4.0:1496184932.058916:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184932.058918:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:4.0:1496184932.058932:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42980 00000100:00000040:4.0:1496184932.058934:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:4.0:1496184932.058937:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104089616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184952 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184932.058944:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104089616:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184932.058950:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184932.058955:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:4.0:1496184932.058958:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184932.058961:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184932 last 1496184931 00000001:00000040:4.0:1496184932.058964:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184932.058966:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184932.058968:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184932.058972:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184932.058975:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184932.058976:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000040:4.0:1496184932.058977:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000040:4.0:1496184932.058978:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184932.058980:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.058984:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184932.058988:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184932.058990:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.058992:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184932.058995:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184932.058997:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184932.058999:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184932.059000:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184932.059008:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089616 00000100:00000040:4.0:1496184932.059012:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.059015:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.059027:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.059029:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104089616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184952 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184932.059035:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089616:12345-10.37.248.124@o2ib1:8 Request procesed in 91us (192us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184932.059040:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42980 00000100:00000040:4.0:1496184932.059041:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184932.059045:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:0.0:1496184932.070115:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:0.0:1496184932.081104:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00100000:4.0:1496184932.094215:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104089648 00000020:00000040:4.0:1496184932.094218:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184932.094219:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184932.094234:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 42981 00000100:00000040:4.0:1496184932.094235:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184932.094238:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdec50 x1568861104089648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184952 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184932.094245:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089648:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184932.094251:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184932.094256:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184932.094261:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184932.094264:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184932 last 1496184931 00000001:00000040:4.0:1496184932.094267:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184932.094269:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184932.094270:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184932.094275:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184932.094278:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184932.094279:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184932.094280:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184932.094281:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184932.094283:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.094287:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184932.094288:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184932.094291:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.094292:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184932.094295:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184932.094297:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184932.094299:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184932.094300:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184932.094304:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104089648 00000100:00000040:4.0:1496184932.094308:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.094310:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.094322:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184932.094324:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdec50 x1568861104089648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184952 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184932.094330:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104089648:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (161us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184932.094336:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 42981 00000100:00000040:4.0:1496184932.094337:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184932.094340:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:1.0:1496184932.097113:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:1.0:1496184932.097139:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184932.127833:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184932.127859:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00020000:3.0:1496184932.158405:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184932.158432:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184932.188726:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.188752:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.219100:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.230256:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.254083:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184932.265146:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184932.289625:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184932.300700:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184932.325145:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184932.336198:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00020000:3.0:1496184932.360924:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00000040:2.0:1496184932.365922:0:6132:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee6050 x1568861152397504 msgsize 224 00000100:00100000:2.0:1496184932.365931:0:6132:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184932.365933:0:6132:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184932.365958:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397504 00000020:00000040:1.0:1496184932.365963:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000040:1.0:1496184932.365967:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000100:00000040:1.0:1496184932.365978:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184932.365996:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 953 00000100:00000040:1.0:1496184932.365999:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000040:1.0:1496184932.366004:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee6050 x1568861152397504/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:683/0 lens 224/0 e 0 to 0 dl 1496184938 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184932.366012:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397504:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184932.366032:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 10831808 pending 1073152 free 80206888960 avail 76094111744 00010000:00000040:1.0:1496184932.366039:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397504 00000100:00000040:1.0:1496184932.366047:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184932.366054:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184932.366076:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184932.366079:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee6050 x1568861152397504/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:683/0 lens 224/336 e 0 to 0 dl 1496184938 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184932.366085:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397504:12345-10.37.248.197@o2ib1:13 Request procesed in 73us (162us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184932.366090:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 953 00000100:00000040:1.0:1496184932.366092:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000020:00000040:1.0:1496184932.366098:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 00000100:00020000:3.0:1496184932.371946:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184932.396709:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.407724:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.431925:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.442943:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184932.454039:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4050 x1568861104090000 msgsize 520 00000100:00100000:1.0:1496184932.454048:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184932.454050:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184932.454070:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090000 00000020:00000040:3.0:1496184932.454074:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184932.454077:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184932.454101:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6009 00000100:00000040:3.0:1496184932.454104:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184932.454109:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4050 x1568861104090000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184952 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184932.454118:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104090000:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184932.454131:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184932.454143:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184932.454148:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184932.454152:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184932 last 1496184931 00000001:00000040:3.0:1496184932.454160:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184932.454163:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184932.454165:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184932.454173:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184932.454177:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.454179:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.454180:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184932.454181:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184932.454184:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.454189:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.454192:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184932.454195:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.454200:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.454205:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184932.454208:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184932.454210:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.454212:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184932.454219:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090000 00000100:00000040:3.0:1496184932.454227:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.454230:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.454249:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.454252:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4050 x1568861104090000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184952 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184932.454258:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090000:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (218us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184932.454263:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6009 00000100:00000040:3.0:1496184932.454265:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.454271:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184932.457092:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.468109:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.495348:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.506370:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.531057:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.542092:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.566211:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.577280:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.601254:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.612285:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.636221:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.647266:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184932.658366:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2c50 x1568861104090192 msgsize 520 00000100:00100000:1.0:1496184932.658375:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184932.658377:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184932.658397:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090192 00000020:00000040:3.0:1496184932.658403:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184932.658407:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184932.658432:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6010 00000100:00000040:3.0:1496184932.658434:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184932.658439:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2c50 x1568861104090192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184952 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184932.658448:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090192:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184932.658460:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184932.658473:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184932.658477:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184932.658481:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184932 last 1496184931 00000001:00000040:3.0:1496184932.658489:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184932.658492:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184932.658494:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184932.658502:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184932.658506:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.658509:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.658510:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184932.658511:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184932.658514:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.658519:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.658521:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184932.658524:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.658526:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.658531:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184932.658535:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184932.658538:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.658540:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184932.658548:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090192 00000100:00000040:3.0:1496184932.658555:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.658559:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.658578:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.658580:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2c50 x1568861104090192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184952 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184932.658586:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090192:12345-10.37.248.124@o2ib1:8 Request procesed in 139us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184932.658591:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6010 00000100:00000040:3.0:1496184932.658593:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.658599:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184932.670301:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.681297:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.705329:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.705356:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.736957:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184932.736968:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184932.767533:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184932.767559:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184932.798405:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.809550:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184932.833569:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.844635:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184932.855773:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3050 x1568861104090384 msgsize 520 00000100:00100000:1.0:1496184932.855782:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184932.855783:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184932.855803:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090384 00000020:00000040:3.0:1496184932.855807:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184932.855810:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184932.855835:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6011 00000100:00000040:3.0:1496184932.855837:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184932.855845:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3050 x1568861104090384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184952 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184932.855854:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104090384:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184932.855867:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184932.855880:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184932.855884:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184932.855888:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184932 last 1496184931 00000001:00000040:3.0:1496184932.855903:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184932.855906:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184932.855908:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184932.855916:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184932.855921:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.855923:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.855924:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184932.855925:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184932.855928:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.855933:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.855936:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184932.855939:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.855941:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184932.855946:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184932.855949:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184932.855951:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184932.855953:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184932.855961:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090384 00000100:00000040:3.0:1496184932.855968:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.855972:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.855991:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184932.855994:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3050 x1568861104090384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184952 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184932.856000:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090384:12345-10.37.248.124@o2ib1:8 Request procesed in 146us (227us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184932.856005:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6011 00000100:00000040:3.0:1496184932.856007:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184932.856013:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184932.858472:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.869486:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184932.894122:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.905178:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184932.929656:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.940708:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.964960:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184932.976008:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184933.000255:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184933.011305:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184933.035176:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184933.046220:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184933.057309:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2050 x1568861104090576 msgsize 520 00000100:00100000:1.0:1496184933.057318:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184933.057319:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184933.057339:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090576 00000020:00000040:3.0:1496184933.057343:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184933.057346:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184933.057371:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6012 00000100:00000040:3.0:1496184933.057373:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184933.057379:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2050 x1568861104090576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184933.057388:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104090576:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.057400:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.057415:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.057420:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.057423:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.057431:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.057434:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.057437:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.057445:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.057449:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.057451:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.057452:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.057454:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.057457:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.057462:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.057464:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.057467:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.057470:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.057475:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.057478:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.057480:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.057482:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.057490:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090576 00000100:00000040:3.0:1496184933.057497:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.057501:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.057520:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.057523:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2050 x1568861104090576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.057529:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090576:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.057536:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6012 00000100:00000040:3.0:1496184933.057537:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.057543:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184933.070402:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184933.081410:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184933.106032:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184933.106043:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184933.137454:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184933.137474:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00020000:3.0:1496184933.167782:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184933.167791:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184933.198401:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184933.209552:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184933.233619:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184933.244660:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00000100:00000040:1.0:1496184933.255795:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1850 x1568861104090784 msgsize 520 00000100:00100000:1.0:1496184933.255804:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184933.255806:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184933.255826:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090784 00000020:00000040:3.0:1496184933.255829:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184933.255833:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184933.255857:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6013 00000100:00000040:3.0:1496184933.255859:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.255865:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1850 x1568861104090784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184933.255874:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104090784:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.255886:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.255904:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.255909:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.255913:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.255921:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.255926:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.255929:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.255937:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.255941:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.255943:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.255944:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.255945:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.255948:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.255954:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.255956:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.255959:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.255961:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.255966:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.255969:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.255972:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.255973:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.255981:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090784 00000100:00000040:3.0:1496184933.255988:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.255992:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.256011:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.256014:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1850 x1568861104090784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.256020:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090784:12345-10.37.248.124@o2ib1:8 Request procesed in 147us (224us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.256025:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6013 00000100:00000040:3.0:1496184933.256027:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.256033:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184933.258834:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184933.269847:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184933.294094:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184933.305122:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184933.329374:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184933.340427:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184933.364866:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184933.375900:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184933.400327:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184933.411369:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184933.435345:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184933.446381:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00000100:00000040:1.0:1496184933.457488:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2850 x1568861104090976 msgsize 520 00000100:00100000:1.0:1496184933.457497:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184933.457499:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184933.457519:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104090976 00000020:00000040:3.0:1496184933.457522:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184933.457526:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184933.457551:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6014 00000100:00000040:3.0:1496184933.457553:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.457558:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2850 x1568861104090976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184933.457567:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104090976:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.457579:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.457592:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.457596:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.457600:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.457608:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.457611:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.457613:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.457622:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.457628:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.457631:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.457632:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.457633:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.457636:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.457641:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.457644:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.457647:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.457649:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.457654:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.457657:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.457659:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.457661:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.457669:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104090976 00000100:00000040:3.0:1496184933.457676:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.457679:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.457698:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.457701:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2850 x1568861104090976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.457707:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104090976:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (218us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.457712:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6014 00000100:00000040:3.0:1496184933.457714:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.457719:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184933.470182:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184933.481199:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184933.505694:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184933.505718:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184933.537297:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184933.537321:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184933.568241:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184933.568267:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184933.598428:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184933.609587:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00000100:00000040:7.0:1496184933.631163:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.631167:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd8c50 x1568861104091136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.631175:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091136:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.631181:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.631186:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.631189:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.631191:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.631195:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.631196:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.631198:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.631202:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.631205:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.631206:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.631207:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.631208:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.631210:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.631214:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.631216:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.631218:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.631220:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.631223:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.631225:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.631227:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.631228:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.631232:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091136 00000100:00000040:7.0:1496184933.631236:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.631239:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.631251:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.631253:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd8c50 x1568861104091136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.631259:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091136:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.631264:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43021 00000100:00000040:7.0:1496184933.631265:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.631268:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.631828:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.631831:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.631832:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.631838:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.631845:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.631852:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.631854:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000100:00100000:7.0:1496184933.631864:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57197 00000100:00000040:7.0:1496184933.631866:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.631868:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.631874:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.631933:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.633399:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184933.644484:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184933.655592:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184933.655619:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1450 x1568861104091168 msgsize 520 00000100:00100000:1.0:1496184933.655627:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184933.655629:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184933.655650:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184933.655652:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091168 00000020:02000400:7.0:1496184933.655655:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184933.655655:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00010000:00000080:7.0:1496184933.655658:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184933.655659:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000040:7.0:1496184933.655666:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.655669:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.655675:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23800us (23892us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.655680:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57197 00000100:00000040:7.0:1496184933.655682:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:3.0:1496184933.655683:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6015 00002000:00100000:7.0:1496184933.655685:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184933.655686:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184933.655691:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00000100:00000040:3.0:1496184933.655691:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1450 x1568861104091168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00010000:7.0:1496184933.655694:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000020:00000040:7.0:1496184933.655698:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184933.655700:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091168:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.655712:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.655725:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.655729:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.655733:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.655741:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.655744:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.655747:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.655755:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.655759:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.655761:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.655762:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.655763:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.655769:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.655774:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.655777:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.655780:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.655782:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.655787:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.655790:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.655792:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.655794:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.655802:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091168 00000100:00000040:3.0:1496184933.655809:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.655812:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.655831:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.655833:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1450 x1568861104091168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.655840:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091168:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (220us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.655845:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6015 00000100:00000040:3.0:1496184933.655847:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.655852:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.656772:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.656776:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.656777:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.656783:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.656791:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.656797:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.656800:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:7.0:1496184933.656818:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57198 00000100:00000040:7.0:1496184933.656820:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.656822:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.656828:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.656872:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.658416:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184933.669433:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184933.680484:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.680541:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.680545:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.680548:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.680554:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.680556:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.680562:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23733us (23836us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.680567:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57198 00000100:00000040:7.0:1496184933.680569:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.680572:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.680578:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.680580:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000020:00000040:7.0:1496184933.680584:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.691678:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091200 00000020:00000040:7.0:1496184933.691681:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.691682:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.691697:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43022 00000100:00000040:7.0:1496184933.691698:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.691701:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0050 x1568861104091200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.691708:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091200:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.691714:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.691722:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.691725:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.691727:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.691731:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.691733:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.691735:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.691739:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.691743:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.691744:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.691745:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.691746:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.691748:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.691752:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.691754:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.691756:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.691758:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.691761:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.691763:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.691765:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.691766:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.691770:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091200 00000100:00000040:7.0:1496184933.691774:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.691777:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.691789:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.691791:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0050 x1568861104091200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.691797:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091200:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (167us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.691802:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43022 00000100:00000040:7.0:1496184933.691804:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.691807:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.692326:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.692330:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.692331:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.692337:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.692345:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.692351:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.692354:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:7.0:1496184933.692363:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57199 00000100:00000040:7.0:1496184933.692365:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.692367:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.692374:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.692416:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.693966:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184933.705007:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184933.716062:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.716118:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.716123:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.716126:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.716131:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.716134:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.716146:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23771us (23863us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.716151:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57199 00000100:00000040:7.0:1496184933.716152:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.716156:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.716161:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.716164:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000020:00000040:7.0:1496184933.716167:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.727096:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091232 00000020:00000040:7.0:1496184933.727099:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.727101:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.727116:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43023 00000100:00000040:7.0:1496184933.727117:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.727120:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9450 x1568861104091232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.727127:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091232:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.727133:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.727139:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.727141:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.727144:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.727148:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.727149:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.727151:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.727155:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.727160:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.727161:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.727162:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.727163:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.727165:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.727169:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.727171:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.727173:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.727175:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.727177:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.727180:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.727182:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.727183:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.727186:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091232 00000100:00000040:7.0:1496184933.727190:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.727193:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.727205:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.727207:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9450 x1568861104091232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.727214:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091232:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.727218:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43023 00000100:00000040:7.0:1496184933.727220:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.727223:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.727816:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.727819:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.727820:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.727826:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.727834:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.727842:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.727845:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000100:00100000:7.0:1496184933.727855:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57200 00000100:00000040:7.0:1496184933.727856:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.727859:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.727865:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.727921:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.729556:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184933.740618:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184933.751680:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.751737:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.751742:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.751745:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.751750:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.751753:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.751759:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23893us (23987us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.751764:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57200 00000100:00000040:7.0:1496184933.751765:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.751769:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.751774:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.751779:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000020:00000040:7.0:1496184933.751783:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.762576:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091264 00000020:00000040:7.0:1496184933.762579:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.762580:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.762595:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43024 00000100:00000040:7.0:1496184933.762597:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.762599:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9050 x1568861104091264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.762607:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091264:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.762612:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.762618:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.762620:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.762623:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.762626:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.762628:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.762630:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.762634:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.762637:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.762638:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.762639:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.762640:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.762642:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.762646:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.762647:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.762650:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.762652:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.762656:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.762659:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.762660:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.762662:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.762665:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091264 00000100:00000040:7.0:1496184933.762669:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.762672:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.762684:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.762686:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9050 x1568861104091264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.762692:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091264:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.762697:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43024 00000100:00000040:7.0:1496184933.762698:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.762701:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.763295:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.763298:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.763300:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.763305:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.763313:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.763319:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.763322:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000100:00100000:7.0:1496184933.763331:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57201 00000100:00000040:7.0:1496184933.763333:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.763335:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.763342:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.763386:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184933.764989:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:0.0:1496184933.776043:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184933.787141:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.787186:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.787191:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.787194:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.787199:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.787201:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.787207:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23865us (23957us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.787212:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57201 00000100:00000040:7.0:1496184933.787214:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.787218:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.787223:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.787226:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000020:00000040:7.0:1496184933.787229:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.788882:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091296 00000020:00000040:7.0:1496184933.788885:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.788887:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.788917:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43025 00000100:00000040:7.0:1496184933.788918:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.788921:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9c50 x1568861104091296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.788937:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091296:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.788945:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.788950:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.788953:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.788955:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.788959:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.788961:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.788962:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.788967:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.788970:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.788971:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.788972:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.788973:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.788975:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.788979:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.788980:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.788982:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.788993:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.788995:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.788998:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.788999:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.789000:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.789004:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091296 00000100:00000040:7.0:1496184933.789008:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.789011:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.789023:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.789025:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9c50 x1568861104091296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.789032:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091296:12345-10.37.248.124@o2ib1:8 Request procesed in 95us (198us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.789037:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43025 00000100:00000040:7.0:1496184933.789038:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.789041:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.789618:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.789621:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.789623:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.789629:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.789637:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.789643:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.789645:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000100:00100000:7.0:1496184933.789655:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57202 00000100:00000040:7.0:1496184933.789657:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.789659:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.789666:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.789709:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.791160:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184933.802184:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184933.813239:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.813297:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.813301:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.813304:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.813309:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.813314:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.813320:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23654us (23746us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.813325:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57202 00000100:00000040:7.0:1496184933.813327:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.813330:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.813336:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.813339:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000020:00000040:7.0:1496184933.813342:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.823707:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091328 00000020:00000040:7.0:1496184933.823715:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.823717:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.823731:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43026 00000100:00000040:7.0:1496184933.823733:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.823736:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9850 x1568861104091328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.823743:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091328:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.823749:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.823754:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.823757:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.823759:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.823763:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.823765:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.823766:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.823773:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.823776:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.823777:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.823778:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.823779:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.823781:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.823785:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.823786:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.823789:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.823790:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.823793:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.823795:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.823797:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.823798:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.823802:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091328 00000100:00000040:7.0:1496184933.823806:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.823808:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.823820:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.823823:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9850 x1568861104091328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.823829:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091328:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (169us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.823833:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43026 00000100:00000040:7.0:1496184933.823835:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.823838:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.824391:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.824394:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.824396:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.824403:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.824411:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.824418:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.824420:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:7.0:1496184933.824430:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57203 00000100:00000040:7.0:1496184933.824432:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.824434:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.824440:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.824483:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.826324:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184933.837383:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00010000:00020000:7.0:1496184933.848454:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.848499:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.848503:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.848506:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.848511:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.848514:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.848520:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24080us (24174us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.848525:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57203 00000100:00000040:7.0:1496184933.848527:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.848530:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.848543:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.848545:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:7.0:1496184933.848548:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.859223:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091360 00000020:00000040:7.0:1496184933.859227:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.859228:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.859242:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43027 00000100:00000040:7.0:1496184933.859244:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.859247:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda850 x1568861104091360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.859254:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091360:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.859260:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.859265:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.859268:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.859270:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.859274:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.859276:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.859277:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.859282:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.859284:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.859286:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.859287:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.859287:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.859289:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.859293:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.859297:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.859299:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.859301:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.859304:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.859306:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.859308:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.859309:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.859312:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091360 00000100:00000040:7.0:1496184933.859316:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.859319:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.859331:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.859333:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda850 x1568861104091360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.859339:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091360:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.859344:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43027 00000100:00000040:7.0:1496184933.859345:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.859348:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.859920:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.859923:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.859925:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.859930:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.859938:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.859944:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.859947:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:7.0:1496184933.859957:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57204 00000100:00000040:7.0:1496184933.859958:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.859963:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.859969:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.860011:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.861645:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184933.872671:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184933.883742:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.883799:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.883803:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.883806:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.883812:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.883814:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.883820:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23851us (23953us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.883825:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57204 00000100:00000040:7.0:1496184933.883827:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.883830:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.883836:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.883839:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000020:00000040:7.0:1496184933.883842:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.894571:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091392 00000020:00000040:7.0:1496184933.894574:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.894576:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.894590:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43028 00000100:00000040:7.0:1496184933.894592:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.894596:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda450 x1568861104091392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.894604:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091392:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.894610:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.894615:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.894618:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.894620:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.894624:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.894625:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.894627:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.894631:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.894634:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.894635:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.894636:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.894637:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.894639:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.894643:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.894645:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.894647:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.894649:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.894651:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.894654:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.894656:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.894657:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.894660:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091392 00000100:00000040:7.0:1496184933.894664:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.894668:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.894680:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.894682:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda450 x1568861104091392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.894688:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091392:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.894692:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43028 00000100:00000040:7.0:1496184933.894694:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.894697:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.895269:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.895272:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.895274:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.895279:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.895287:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.895294:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.895296:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:7.0:1496184933.895306:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57205 00000100:00000040:7.0:1496184933.895307:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.895310:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.895316:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.895359:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.897228:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f400 00000100:00020000:3.0:1496184933.897239:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f400 00010000:00020000:7.0:1496184933.897283:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.897329:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.897333:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.897336:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.897349:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.897352:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.897358:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2041us (2139us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.897362:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57205 00000100:00000040:7.0:1496184933.897364:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.897368:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.897373:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.897376:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:7.0:1496184933.897379:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:0.0:1496184933.925619:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1c50 x1568861104091424 msgsize 520 00000100:00100000:0.0:1496184933.925628:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184933.925630:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184933.925677:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091424 00000020:00000040:3.0:1496184933.925682:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184933.925686:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184933.925713:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6016 00000100:00000040:3.0:1496184933.925715:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.925720:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1c50 x1568861104091424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184933.925729:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091424:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.925741:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.925752:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.925757:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.925760:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.925768:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.925772:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.925774:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.925782:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.925786:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.925788:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.925789:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.925790:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.925793:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.925799:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.925804:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.925807:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.925809:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.925815:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.925818:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.925820:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.925822:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.925830:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091424 00000100:00000040:3.0:1496184933.925837:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.925840:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.925863:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.925866:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1c50 x1568861104091424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.925872:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091424:12345-10.37.248.124@o2ib1:8 Request procesed in 144us (252us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.925877:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6016 00000100:00000040:3.0:1496184933.925879:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.925884:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.926437:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.926440:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.926442:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.926448:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.926456:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.926463:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.926465:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000100:00100000:7.0:1496184933.926475:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57206 00000100:00000040:7.0:1496184933.926476:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.926479:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.926486:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.926529:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.927828:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f400 00000100:00020000:3.0:1496184933.927852:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f400 00010000:00020000:7.0:1496184933.927905:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.927959:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.927964:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.927967:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.927972:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.927974:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.927980:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1495us (1588us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.927985:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57206 00000100:00000040:7.0:1496184933.927987:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.927991:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.927996:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.927999:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000020:00000040:7.0:1496184933.928002:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:0.0:1496184933.956352:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1050 x1568861104091456 msgsize 520 00000100:00100000:0.0:1496184933.956360:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184933.956361:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184933.956401:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091456 00000020:00000040:3.0:1496184933.956405:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184933.956407:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184933.956422:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6017 00000100:00000040:3.0:1496184933.956424:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.956427:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1050 x1568861104091456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184933.956434:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091456:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184933.956445:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184933.956451:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184933.956453:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184933.956456:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:3.0:1496184933.956461:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184933.956463:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184933.956464:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184933.956469:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184933.956472:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.956474:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.956475:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184933.956476:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184933.956478:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.956482:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.956483:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184933.956486:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.956488:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184933.956490:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184933.956492:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184933.956494:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184933.956495:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184933.956499:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091456 00000100:00000040:3.0:1496184933.956504:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.956506:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.956519:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184933.956521:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1050 x1568861104091456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184933.956529:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091456:12345-10.37.248.124@o2ib1:8 Request procesed in 95us (175us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184933.956533:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6017 00000100:00000040:3.0:1496184933.956535:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184933.956538:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.957074:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.957077:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.957079:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.957085:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.957093:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.957099:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.957102:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000100:00100000:7.0:1496184933.957112:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57207 00000100:00000040:7.0:1496184933.957115:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.957118:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.957124:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.957167:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.958553:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f400 00000100:00020000:3.0:1496184933.958576:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f400 00010000:00020000:7.0:1496184933.958620:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.958665:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.958670:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.958673:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.958678:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.958681:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.958686:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1562us (1658us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.958691:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57207 00000100:00000040:7.0:1496184933.958693:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.958697:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.958702:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.958705:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000020:00000040:7.0:1496184933.958708:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.987309:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091488 00000020:00000040:7.0:1496184933.987312:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.987313:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184933.987330:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43029 00000100:00000040:7.0:1496184933.987331:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.987334:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdac50 x1568861104091488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184953 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.987341:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091488:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184933.987348:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184933.987353:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184933.987356:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184933.987358:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184933 last 1496184933 00000001:00000040:7.0:1496184933.987362:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184933.987364:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184933.987366:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184933.987370:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184933.987373:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.987375:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.987376:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184933.987377:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.987379:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.987383:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.987385:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184933.987387:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.987389:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184933.987392:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184933.987394:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184933.987396:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184933.987397:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184933.987402:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091488 00000100:00000040:7.0:1496184933.987406:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.987409:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.987420:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184933.987423:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdac50 x1568861104091488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184953 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184933.987429:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091488:12345-10.37.248.124@o2ib1:8 Request procesed in 88us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184933.987433:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43029 00000100:00000040:7.0:1496184933.987435:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184933.987438:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184933.987901:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184933.987904:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184933.987905:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184933.987911:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184933.987918:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184933.987925:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.987927:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000100:00100000:7.0:1496184933.987937:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57208 00000100:00000040:7.0:1496184933.987939:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184933.987941:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/0 e 0 to 0 dl 1496184976 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184933.987948:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184933.987990:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184933.989595:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184933.989619:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184933.989664:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184933.989718:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184933.989724:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184933.989728:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184933.989733:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184933.989735:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184933.989741:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1793us (1894us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184933.989746:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57208 00000100:00000040:7.0:1496184933.989748:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184933.989751:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:721/0 lens 608/448 e 0 to 0 dl 1496184976 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184933.989757:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184933.989759:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000020:00000040:7.0:1496184933.989762:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.017857:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091520 00000020:00000040:7.0:1496184934.017865:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.017866:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.017881:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43030 00000100:00000040:7.0:1496184934.017883:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.017885:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb050 x1568861104091520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.017904:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091520:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.017910:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.017915:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.017918:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.017928:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.017934:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.017936:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.017937:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.017943:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.017946:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.017948:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.017949:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.017950:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.017952:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.017956:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.017957:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.017959:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.017961:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.017964:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.017966:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.017968:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.017969:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.017973:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091520 00000100:00000040:7.0:1496184934.017977:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.017979:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.017994:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.017996:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb050 x1568861104091520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.018003:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091520:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (191us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.018007:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43030 00000100:00000040:7.0:1496184934.018008:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.018012:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.018612:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.018616:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.018617:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.018622:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.018630:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.018637:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.018639:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000100:00100000:7.0:1496184934.018649:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57209 00000100:00000040:7.0:1496184934.018651:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.018653:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.018660:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.018702:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.020149:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184934.031314:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184934.042388:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.042446:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.042451:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.042454:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.042459:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.042462:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.042468:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23808us (23901us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.042473:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57209 00000100:00000040:7.0:1496184934.042477:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.042480:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.042486:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.042489:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000020:00000040:7.0:1496184934.042492:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.052941:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091552 00000020:00000040:7.0:1496184934.052944:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.052945:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.052960:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43031 00000100:00000040:7.0:1496184934.052961:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.052964:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdbc50 x1568861104091552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.052971:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091552:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.052977:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.052982:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.052985:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.052988:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.052991:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.052993:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.052994:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.052999:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.053002:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.053003:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.053004:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.053005:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.053009:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.053013:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.053014:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.053017:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.053019:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.053021:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.053023:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.053025:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.053026:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.053030:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091552 00000100:00000040:7.0:1496184934.053034:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.053037:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.053048:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.053051:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdbc50 x1568861104091552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.053057:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091552:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (173us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.053061:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43031 00000100:00000040:7.0:1496184934.053063:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.053066:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.053730:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.053733:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.053735:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.053741:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.053749:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.053755:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.053758:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000100:00100000:7.0:1496184934.053769:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57210 00000100:00000040:7.0:1496184934.053771:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.053773:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.053780:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.053820:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.055646:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184934.066704:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184934.077775:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.077833:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.077837:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.077840:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.077845:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.077848:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.077853:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24073us (24175us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.077858:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57210 00000100:00000040:7.0:1496184934.077860:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.077864:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.077869:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.077872:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000020:00000040:7.0:1496184934.077875:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.088238:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091584 00000020:00000040:7.0:1496184934.088241:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.088245:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.088259:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43032 00000100:00000040:7.0:1496184934.088261:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.088263:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb850 x1568861104091584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.088270:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091584:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.088276:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.088282:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.088284:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.088287:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.088291:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.088292:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.088294:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.088298:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.088301:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.088303:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.088303:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.088304:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.088306:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.088310:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.088312:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.088314:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.088316:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.088318:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.088320:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.088322:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.088324:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.088328:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091584 00000100:00000040:7.0:1496184934.088332:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.088334:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.088346:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.088349:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb850 x1568861104091584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.088355:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091584:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.088359:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43032 00000100:00000040:7.0:1496184934.088360:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.088364:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.088903:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.088906:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.088908:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.088913:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.088921:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.088928:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.088930:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000100:00100000:7.0:1496184934.088940:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57211 00000100:00000040:7.0:1496184934.088942:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.088944:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.088951:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.088993:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.090674:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184934.101746:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184934.112797:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.112860:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.112865:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.112868:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.112873:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.112875:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.112881:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23930us (24026us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.112886:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57211 00000100:00000040:7.0:1496184934.112888:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.112904:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.112910:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.112912:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000020:00000040:7.0:1496184934.112916:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.123911:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091616 00000020:00000040:7.0:1496184934.123915:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.123916:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.123931:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43033 00000100:00000040:7.0:1496184934.123936:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.123938:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb450 x1568861104091616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.123946:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091616:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.123951:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.123959:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.123961:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.123964:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.123968:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.123969:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.123971:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.123975:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.123978:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.123979:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.123980:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.123981:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.123983:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.123987:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.123996:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.123998:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.124000:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.124003:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.124005:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.124007:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.124008:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.124012:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091616 00000100:00000040:7.0:1496184934.124016:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.124018:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.124030:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.124033:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb450 x1568861104091616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.124039:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091616:12345-10.37.248.124@o2ib1:8 Request procesed in 93us (181us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.124044:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43033 00000100:00000040:7.0:1496184934.124045:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.124049:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.124642:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.124645:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.124646:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.124652:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.124660:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.124666:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.124669:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000100:00100000:7.0:1496184934.124678:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57212 00000100:00000040:7.0:1496184934.124680:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.124682:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.124689:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.124730:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.126514:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184934.137561:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184934.148609:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.148666:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.148671:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.148674:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.148679:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.148681:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.148693:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24004us (24096us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.148698:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57212 00000100:00000040:7.0:1496184934.148700:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.148703:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.148709:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.148712:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000020:00000040:7.0:1496184934.148715:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.159352:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091648 00000020:00000040:7.0:1496184934.159355:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.159357:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.159371:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43034 00000100:00000040:7.0:1496184934.159373:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.159375:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc050 x1568861104091648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.159383:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091648:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.159389:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.159394:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.159396:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.159399:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.159403:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.159404:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.159406:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.159410:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.159415:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.159417:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.159417:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.159418:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.159420:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.159424:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.159426:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.159428:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.159430:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.159433:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.159435:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.159437:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.159438:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.159441:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091648 00000100:00000040:7.0:1496184934.159446:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.159448:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.159460:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.159463:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc050 x1568861104091648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.159469:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091648:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.159473:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43034 00000100:00000040:7.0:1496184934.159475:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.159478:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.160081:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.160084:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.160086:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.160091:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.160099:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.160107:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.160110:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000100:00100000:7.0:1496184934.160120:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57213 00000100:00000040:7.0:1496184934.160121:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.160124:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.160130:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.160172:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.161805:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184934.172855:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184934.183919:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.183978:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.183982:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.183985:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.183990:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.183993:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.183999:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23869us (23964us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.184004:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57213 00000100:00000040:7.0:1496184934.184005:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.184009:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.184015:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.184019:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000020:00000040:7.0:1496184934.184023:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.194671:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091680 00000020:00000040:7.0:1496184934.194674:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.194676:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.194690:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43035 00000100:00000040:7.0:1496184934.194691:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.194694:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc450 x1568861104091680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.194702:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091680:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.194707:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.194713:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.194715:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.194718:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.194721:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.194723:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.194724:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.194729:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.194732:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.194733:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.194734:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.194735:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.194737:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.194741:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.194742:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.194745:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.194747:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.194751:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.194754:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.194755:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.194757:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.194760:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091680 00000100:00000040:7.0:1496184934.194764:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.194767:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.194779:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.194781:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc450 x1568861104091680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.194787:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091680:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.194792:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43035 00000100:00000040:7.0:1496184934.194793:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.194796:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.195411:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.195414:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.195416:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.195422:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.195430:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.195436:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.195439:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000100:00100000:7.0:1496184934.195449:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57214 00000100:00000040:7.0:1496184934.195450:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.195453:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.195459:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.195504:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.197190:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184934.208229:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00010000:00020000:7.0:1496184934.219289:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.219347:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.219352:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.219355:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.219360:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.219363:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.219368:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23909us (24003us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.219374:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57214 00000100:00000040:7.0:1496184934.219375:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.219379:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.219385:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.219387:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000020:00000040:7.0:1496184934.219390:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.230084:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091712 00000020:00000040:7.0:1496184934.230087:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.230089:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.230103:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43036 00000100:00000040:7.0:1496184934.230105:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.230108:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc850 x1568861104091712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.230115:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091712:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.230123:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.230128:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.230131:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.230133:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.230137:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.230139:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.230140:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.230144:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.230147:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.230149:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.230150:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.230150:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.230152:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.230156:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.230158:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.230160:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.230162:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.230164:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.230167:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.230168:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.230169:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.230173:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091712 00000100:00000040:7.0:1496184934.230177:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.230180:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.230192:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.230194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc850 x1568861104091712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.230201:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091712:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.230205:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43036 00000100:00000040:7.0:1496184934.230207:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.230210:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.230772:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.230775:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.230776:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.230782:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.230790:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.230796:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.230799:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000100:00100000:7.0:1496184934.230808:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57215 00000100:00000040:7.0:1496184934.230809:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.230812:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.230818:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.230861:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.232457:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184934.243500:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00010000:00020000:7.0:1496184934.254579:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184934.254605:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0450 x1568861104091744 msgsize 520 00000100:00100000:1.0:1496184934.254614:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184934.254616:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184934.254636:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091744 00000020:00000040:7.0:1496184934.254637:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.254640:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:02000400:7.0:1496184934.254642:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184934.254643:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000080:7.0:1496184934.254645:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.254650:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.254655:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.254661:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23842us (23942us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.254666:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57215 00000100:00000040:7.0:1496184934.254668:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:3.0:1496184934.254668:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6018 00000100:00000040:3.0:1496184934.254671:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184934.254672:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184934.254676:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0450 x1568861104091744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.254678:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.254680:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000020:00000040:7.0:1496184934.254684:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184934.254691:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091744:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184934.254704:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184934.254716:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184934.254720:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184934.254724:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:3.0:1496184934.254733:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184934.254736:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184934.254739:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184934.254747:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184934.254751:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.254753:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.254754:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184934.254755:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184934.254758:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.254764:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.254766:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184934.254769:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.254771:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.254777:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184934.254780:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184934.254782:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.254784:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184934.254792:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091744 00000100:00000040:3.0:1496184934.254799:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.254803:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.254822:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.254825:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0450 x1568861104091744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184934.254831:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091744:12345-10.37.248.124@o2ib1:8 Request procesed in 141us (225us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184934.254836:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6018 00000100:00000040:3.0:1496184934.254838:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.254843:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.255751:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.255755:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.255757:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.255762:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.255777:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.255784:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.255786:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000100:00100000:7.0:1496184934.255796:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57216 00000100:00000040:7.0:1496184934.255798:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.255800:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.255807:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.255850:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.257240:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184934.268250:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00010000:00020000:7.0:1496184934.279311:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.279374:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.279379:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.279382:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.279387:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.279389:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.279403:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23588us (23690us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.279408:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57216 00000100:00000040:7.0:1496184934.279410:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.279413:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.279419:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.279421:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000020:00000040:7.0:1496184934.279425:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.291537:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091776 00000020:00000040:7.0:1496184934.291541:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.291542:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.291557:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43037 00000100:00000040:7.0:1496184934.291558:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.291561:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdcc50 x1568861104091776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.291568:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091776:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.291574:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.291582:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.291584:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.291587:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.291591:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.291593:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.291594:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.291599:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.291602:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.291603:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.291604:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.291605:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.291607:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.291612:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.291613:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.291616:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.291618:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.291620:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.291623:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.291625:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.291626:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.291629:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091776 00000100:00000040:7.0:1496184934.291633:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.291636:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.291648:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.291650:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdcc50 x1568861104091776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.291657:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091776:12345-10.37.248.124@o2ib1:8 Request procesed in 88us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.291662:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43037 00000100:00000040:7.0:1496184934.291663:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.291666:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.292204:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.292207:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.292209:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.292215:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.292223:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.292229:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.292232:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000100:00100000:7.0:1496184934.292242:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57217 00000100:00000040:7.0:1496184934.292243:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.292246:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.292252:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.292295:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.293957:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184934.304985:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184934.316040:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.316097:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.316102:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.316105:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.316110:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.316112:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.316124:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23872us (23976us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.316130:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57217 00000100:00000040:7.0:1496184934.316131:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.316135:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.316141:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.316143:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000020:00000040:7.0:1496184934.316146:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.326903:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091808 00000020:00000040:7.0:1496184934.326906:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.326907:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.326922:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43038 00000100:00000040:7.0:1496184934.326924:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.326927:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd050 x1568861104091808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.326934:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091808:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.326940:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.326945:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.326948:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.326951:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.326954:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.326956:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.326957:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.326962:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.326967:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.326968:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.326969:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.326970:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.326972:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.326976:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.326977:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.326980:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.326981:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.326984:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.326986:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.326988:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.326989:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.326993:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091808 00000100:00000040:7.0:1496184934.326997:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.326999:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.327012:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.327014:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd050 x1568861104091808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.327020:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091808:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (169us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.327025:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43038 00000100:00000040:7.0:1496184934.327026:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.327030:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.327612:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.327615:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.327617:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.327622:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.327630:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.327639:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.327641:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000100:00100000:7.0:1496184934.327651:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57218 00000100:00000040:7.0:1496184934.327652:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.327655:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.327661:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.327704:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.329451:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184934.340495:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00000100:00000040:2.0:1496184934.340917:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184934.340928:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184934.340948:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.340960:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76a700] to pc [ptlrpcd_00_02:2] req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.340969:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184934.340976:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184934.340979:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.340985:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f769500] to pc [ptlrpcd_00_03:3] req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.340990:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184934.340992:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184934.340996:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.341001:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76a100] to pc [ptlrpcd_00_00:0] req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184934.341007:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184934.341007:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.341011:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 5000 (4297373369) 00000100:00100000:2.0:1496184934.341017:0:13690:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1496184934.341021:0:13692:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:13692:1568861159710160:10.37.248.197@o2ib1:400 00000100:00100000:0.0:1496184934.341021:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:d86c02e9-1939-02b8-4331-d3612fcf0106:13691:1568861159710144:10.37.248.196@o2ib1:400 00000100:00000040:2.0:1496184934.341022:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184934.341029:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710176:10.37.248.197@o2ib1:400 00000100:00000040:2.0:1496184934.341041:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184934.341041:0:13692:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184934.341041:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.341048:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184934.341048:0:13692:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:0.0:1496184934.341048:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:2.0:1496184934.341121:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.341130:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.341405:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184934.341437:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184934.341462:0:13690:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:1.0:1496184934.341468:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184934.341476:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:1.0:1496184934.341482:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710176:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184934.341486:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76a100 x1568861159710176/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184934.341490:0:13692:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000020:00000040:1.0:1496184934.341492:0:13690:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184934.341496:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184934.341506:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:2.0:1496184934.341511:0:13692:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:lustre-MDT0000-lwp-OST0000_UUID:13692:1568861159710160:10.37.248.197@o2ib1:400 00000100:00000040:2.0:1496184934.341515:0:13692:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f769500 x1568861159710160/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:2.0:1496184934.341520:0:13692:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184934.351546:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184941 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00010000:00020000:7.0:1496184934.351559:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:3.0:1496184934.351559:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184934.351591:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184934.351597:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:2.0:1496184934.351602:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:d86c02e9-1939-02b8-4331-d3612fcf0106:13691:1568861159710144:10.37.248.196@o2ib1:400 00000100:00000040:2.0:1496184934.351606:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76a700 x1568861159710144/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184941 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:2.0:1496184934.351611:0:13691:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000020:00000040:7.0:1496184934.351619:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.351624:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.351627:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.351632:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.351635:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.351640:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23979us (24071us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.351645:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57218 00000100:00000040:7.0:1496184934.351647:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.351651:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.351656:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.351666:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000020:00000040:7.0:1496184934.351670:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.362385:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091840 00000020:00000040:7.0:1496184934.362389:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.362390:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.362405:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43039 00000100:00000040:7.0:1496184934.362407:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.362409:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd450 x1568861104091840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.362417:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091840:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.362422:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.362427:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.362430:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.362433:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.362436:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.362438:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.362440:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.362444:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.362447:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.362448:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.362449:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.362450:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.362452:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.362456:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.362457:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.362460:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.362461:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.362466:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.362468:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.362470:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.362471:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.362475:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091840 00000100:00000040:7.0:1496184934.362479:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.362482:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.362494:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.362496:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd450 x1568861104091840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.362502:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091840:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.362507:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43039 00000100:00000040:7.0:1496184934.362508:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.362512:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.363109:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.363112:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.363114:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.363120:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.363128:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.363134:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.363137:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000100:00100000:7.0:1496184934.363147:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57219 00000100:00000040:7.0:1496184934.363148:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.363150:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.363157:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.363201:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.364901:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184934.375913:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184934.386979:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.387037:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.387042:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.387045:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.387050:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.387053:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.387059:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23901us (23994us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.387063:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57219 00000100:00000040:7.0:1496184934.387065:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.387069:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.387074:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.387077:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000020:00000040:7.0:1496184934.387080:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.398084:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091872 00000020:00000040:7.0:1496184934.398087:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.398089:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.398103:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43040 00000100:00000040:7.0:1496184934.398105:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.398108:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd850 x1568861104091872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.398115:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091872:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.398123:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.398128:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.398131:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.398133:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.398137:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.398139:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.398140:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.398144:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.398147:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.398149:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.398150:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.398150:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.398152:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.398156:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.398158:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.398160:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.398162:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.398164:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.398167:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.398168:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.398169:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.398173:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091872 00000100:00000040:7.0:1496184934.398177:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.398180:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.398192:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.398194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd850 x1568861104091872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.398201:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091872:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (170us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.398205:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43040 00000100:00000040:7.0:1496184934.398207:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.398210:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.398819:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.398823:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.398824:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.398829:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.398837:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.398844:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.398846:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000100:00100000:7.0:1496184934.398856:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57220 00000100:00000040:7.0:1496184934.398858:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.398860:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.398867:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.398924:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.400756:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184934.411793:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184934.422873:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.422928:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.422933:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.422936:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.422941:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.422946:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.422952:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24085us (24182us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.422957:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57220 00000100:00000040:7.0:1496184934.422958:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.422962:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.422968:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.422970:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000020:00000040:7.0:1496184934.422974:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.433504:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091904 00000020:00000040:7.0:1496184934.433507:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.433509:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.433523:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43041 00000100:00000040:7.0:1496184934.433525:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.433528:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104091904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.433535:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091904:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.433541:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.433546:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.433549:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.433551:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.433555:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.433557:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.433558:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.433565:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.433568:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.433569:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.433570:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.433571:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.433573:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.433577:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.433578:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.433580:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.433582:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.433585:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.433587:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.433589:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.433590:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.433593:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091904 00000100:00000040:7.0:1496184934.433598:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.433600:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.433612:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.433615:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104091904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.433621:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091904:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (170us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.433625:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43041 00000100:00000040:7.0:1496184934.433627:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.433630:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.434232:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.434235:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.434237:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.434244:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.434252:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.434259:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.434261:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000100:00100000:7.0:1496184934.434271:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57221 00000100:00000040:7.0:1496184934.434273:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.434275:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.434281:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.434324:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.435862:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184934.446910:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184934.457998:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184934.458018:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0850 x1568861104091936 msgsize 520 00000100:00100000:1.0:1496184934.458027:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184934.458029:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184934.458049:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091936 00000020:00000040:3.0:1496184934.458053:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.458056:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.458057:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:02000400:7.0:1496184934.458061:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.458064:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.458069:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.458072:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.458078:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23796us (23897us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184934.458082:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6019 00000100:00100000:7.0:1496184934.458083:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57221 00000100:00000040:3.0:1496184934.458084:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:7.0:1496184934.458085:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184934.458089:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184934.458090:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0850 x1568861104091936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184934.458098:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104091936:12345-10.37.248.124@o2ib1:8 00002000:00100000:7.0:1496184934.458101:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.458104:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000020:00000040:7.0:1496184934.458108:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.458113:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184934.458126:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184934.458131:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184934.458134:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:3.0:1496184934.458142:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184934.458146:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184934.458148:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184934.458156:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184934.458160:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.458162:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.458163:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184934.458164:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184934.458167:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.458173:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.458175:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184934.458179:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.458181:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.458185:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184934.458188:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184934.458191:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.458192:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184934.458200:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091936 00000100:00000040:3.0:1496184934.458207:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.458211:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.458230:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.458233:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0850 x1568861104091936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184934.458240:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091936:12345-10.37.248.124@o2ib1:8 Request procesed in 142us (221us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184934.458245:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6019 00000100:00000040:3.0:1496184934.458247:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.458252:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.459222:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.459225:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.459227:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.459233:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.459248:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.459254:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.459257:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000100:00100000:7.0:1496184934.459266:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57222 00000100:00000040:7.0:1496184934.459268:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.459270:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.459277:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.459320:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.470304:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184934.481303:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184934.492355:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.492413:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.492418:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.492421:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.492426:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.492428:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.492440:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33163us (33261us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.492445:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57222 00000100:00000040:7.0:1496184934.492447:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.492451:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.492456:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.492459:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000020:00000040:7.0:1496184934.492462:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.502876:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104091968 00000020:00000040:7.0:1496184934.502879:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.502881:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.502907:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43042 00000100:00000040:7.0:1496184934.502909:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.502912:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104091968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.502919:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091968:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.502925:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.502931:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.502933:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.502936:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.502940:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.502942:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.502943:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.502948:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.502953:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.502954:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.502955:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.502956:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.502959:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.502963:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.502964:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.502967:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.502969:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.502971:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.502974:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.502976:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.502977:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.502981:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104091968 00000100:00000040:7.0:1496184934.502985:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.502987:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.503004:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.503006:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104091968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.503013:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104091968:12345-10.37.248.124@o2ib1:8 Request procesed in 93us (183us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.503017:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43042 00000100:00000040:7.0:1496184934.503018:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.503022:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.503578:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.503581:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.503583:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.503588:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.503596:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.503604:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.503607:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000100:00100000:7.0:1496184934.503617:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57223 00000100:00000040:7.0:1496184934.503618:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.503621:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.503627:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.503670:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.505656:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184934.505681:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90cc00 00010000:00020000:7.0:1496184934.505724:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.505769:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.505773:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.505777:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.505781:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.505784:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.505790:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2162us (2256us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.505795:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57223 00000100:00000040:7.0:1496184934.505797:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.505800:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.505806:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.505810:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000020:00000040:7.0:1496184934.505814:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.534032:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092000 00000020:00000040:7.0:1496184934.534035:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.534036:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.534051:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43043 00000100:00000040:7.0:1496184934.534053:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.534055:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104092000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.534062:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092000:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.534068:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.534073:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.534076:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.534078:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.534082:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.534084:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.534085:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.534090:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.534093:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.534094:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.534095:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.534096:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.534098:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.534102:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.534103:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.534105:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.534107:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.534112:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.534114:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.534116:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.534117:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.534121:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092000 00000100:00000040:7.0:1496184934.534125:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.534127:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.534139:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.534142:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104092000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.534148:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092000:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.534152:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43043 00000100:00000040:7.0:1496184934.534153:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.534157:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.534687:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.534690:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.534692:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.534697:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.534705:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.534711:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.534713:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000100:00100000:7.0:1496184934.534723:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57224 00000100:00000040:7.0:1496184934.534725:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.534727:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.534734:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.534777:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.536557:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e200 00000100:00020000:3.0:1496184934.536580:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e200 00010000:00020000:7.0:1496184934.536623:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.536677:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.536681:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.536684:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.536689:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.536692:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.536698:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1964us (2056us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.536703:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57224 00000100:00000040:7.0:1496184934.536705:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.536708:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.536714:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.536716:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000020:00000040:7.0:1496184934.536719:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.564858:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092032 00000020:00000040:7.0:1496184934.564861:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.564863:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.564877:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43044 00000100:00000040:7.0:1496184934.564879:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.564882:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104092032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.564889:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092032:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.564910:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.564915:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.564918:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.564920:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.564932:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.564934:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.564935:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.564939:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.564942:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.564943:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.564944:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.564945:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.564947:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.564951:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.564953:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.564955:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.564957:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.564959:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.564962:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.564963:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.564964:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.564968:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092032 00000100:00000040:7.0:1496184934.564972:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.564975:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.564995:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.564998:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104092032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.565005:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092032:12345-10.37.248.124@o2ib1:8 Request procesed in 116us (191us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.565009:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43044 00000100:00000040:7.0:1496184934.565011:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.565014:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.565576:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.565579:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.565580:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.565586:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.565594:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.565601:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.565603:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000100:00100000:7.0:1496184934.565613:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57225 00000100:00000040:7.0:1496184934.565614:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.565617:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.565623:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.565667:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.567243:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184934.567268:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184934.567311:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.567363:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.567367:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.567370:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.567375:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.567380:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.567386:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1763us (1853us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.567391:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57225 00000100:00000040:7.0:1496184934.567393:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.567396:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.567402:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.567405:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000020:00000040:7.0:1496184934.567408:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.595062:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092080 00000020:00000040:7.0:1496184934.595066:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.595069:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.595096:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43045 00000100:00000040:7.0:1496184934.595098:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.595100:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdec50 x1568861104092080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.595108:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092080:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.595113:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.595119:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.595121:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.595124:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.595128:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.595129:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.595131:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.595137:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.595140:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.595141:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.595142:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.595143:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.595145:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.595149:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.595150:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.595153:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.595154:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.595157:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.595159:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.595161:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.595162:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.595166:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092080 00000100:00000040:7.0:1496184934.595170:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.595172:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.595184:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.595187:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdec50 x1568861104092080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.595193:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092080:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (191us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.595197:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43045 00000100:00000040:7.0:1496184934.595199:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.595202:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.595824:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.595828:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.595829:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.595836:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.595844:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.595851:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.595853:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000100:00100000:7.0:1496184934.595863:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57226 00000100:00000040:7.0:1496184934.595865:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.595867:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.595874:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.595932:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.597481:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184934.608628:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184934.619684:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.619742:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.619746:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.619750:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.619755:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.619757:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.619763:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23889us (23990us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.619768:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57226 00000100:00000040:7.0:1496184934.619770:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.619773:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.619784:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.619787:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000020:00000040:7.0:1496184934.619790:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.630205:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092112 00000020:00000040:7.0:1496184934.630208:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.630210:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.630224:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43046 00000100:00000040:7.0:1496184934.630226:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.630229:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf050 x1568861104092112/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.630236:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092112:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.630242:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.630247:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.630250:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.630252:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.630256:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.630258:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.630259:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.630263:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.630266:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.630268:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.630268:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.630269:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.630271:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.630275:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.630277:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.630279:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.630281:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.630284:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.630286:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.630288:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.630289:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.630292:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092112 00000100:00000040:7.0:1496184934.630297:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.630299:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.630311:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.630314:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf050 x1568861104092112/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.630320:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092112:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (157us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.630324:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43046 00000100:00000040:7.0:1496184934.630326:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.630329:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.630907:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.630910:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.630912:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.630917:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.630925:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.630931:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.630934:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000100:00100000:7.0:1496184934.630943:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57227 00000100:00000040:7.0:1496184934.630945:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.630948:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.630954:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.630998:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.632422:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184934.643483:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184934.654612:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184934.654627:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0c50 x1568861104092144 msgsize 520 00000100:00100000:1.0:1496184934.654636:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184934.654637:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184934.654658:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184934.654658:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092144 00000020:00000040:3.0:1496184934.654661:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:02000400:7.0:1496184934.654662:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184934.654665:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000080:7.0:1496184934.654666:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.654671:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.654673:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.654679:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23725us (23830us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.654684:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57227 00000100:00000040:7.0:1496184934.654686:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.654690:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184934.654690:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6020 00000100:00000040:3.0:1496184934.654693:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184934.654696:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00000100:00000040:3.0:1496184934.654698:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0c50 x1568861104092144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00010000:7.0:1496184934.654699:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000020:00000040:7.0:1496184934.654703:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184934.654707:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092144:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184934.654719:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184934.654732:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184934.654736:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184934.654742:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:3.0:1496184934.654751:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184934.654754:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184934.654756:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184934.654765:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184934.654769:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.654771:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.654772:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184934.654773:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184934.654776:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.654781:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.654784:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184934.654787:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.654789:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184934.654794:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184934.654797:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184934.654799:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184934.654801:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184934.654809:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092144 00000100:00000040:3.0:1496184934.654816:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.654820:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.654837:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184934.654840:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0c50 x1568861104092144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184934.654846:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092144:12345-10.37.248.124@o2ib1:8 Request procesed in 140us (219us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184934.654854:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6020 00000100:00000040:3.0:1496184934.654856:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184934.654861:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.655826:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.655830:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.655831:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.655837:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.655845:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.655852:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.655854:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000100:00100000:7.0:1496184934.655864:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57228 00000100:00000040:7.0:1496184934.655866:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.655868:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.655874:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.655928:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.657655:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184934.668676:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184934.679711:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.679769:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.679773:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.679776:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.679782:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.679784:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.679790:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23915us (24009us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.679795:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57228 00000100:00000040:7.0:1496184934.679797:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.679800:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.679806:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.679809:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000020:00000040:7.0:1496184934.679813:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.693260:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092176 00000020:00000040:7.0:1496184934.693263:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.693265:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.693280:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43047 00000100:00000040:7.0:1496184934.693281:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.693284:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf450 x1568861104092176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.693291:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092176:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.693297:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.693302:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.693305:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.693308:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.693312:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.693314:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.693315:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.693320:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.693323:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.693324:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.693325:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.693326:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.693328:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.693332:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.693334:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.693336:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.693338:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.693341:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.693344:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.693346:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.693347:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.693350:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092176 00000100:00000040:7.0:1496184934.693355:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.693357:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.693369:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.693372:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf450 x1568861104092176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.693378:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092176:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.693382:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43047 00000100:00000040:7.0:1496184934.693384:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.693387:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.693911:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.693914:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.693916:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.693922:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.693929:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.693936:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.693939:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000100:00100000:7.0:1496184934.693948:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57229 00000100:00000040:7.0:1496184934.693950:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.693952:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.693959:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.694001:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.695414:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184934.706445:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184934.717497:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.717556:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.717560:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.717563:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.717569:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.717571:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.717577:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23618us (23710us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.717582:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57229 00000100:00000040:7.0:1496184934.717584:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.717587:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.717593:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.717596:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000020:00000040:7.0:1496184934.717599:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.728331:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092208 00000020:00000040:7.0:1496184934.728334:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.728336:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.728350:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43048 00000100:00000040:7.0:1496184934.728352:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.728355:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf850 x1568861104092208/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.728362:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092208:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.728368:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.728373:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.728376:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.728378:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.728382:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.728384:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.728385:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.728390:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.728393:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.728394:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.728395:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.728395:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.728397:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.728401:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.728403:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.728405:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.728407:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.728409:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.728412:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.728413:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.728415:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.728418:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092208 00000100:00000040:7.0:1496184934.728422:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.728425:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.728437:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.728439:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf850 x1568861104092208/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.728445:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092208:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (157us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.728450:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43048 00000100:00000040:7.0:1496184934.728451:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.728454:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.729058:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.729061:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.729063:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.729068:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.729076:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.729082:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.729085:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000100:00100000:7.0:1496184934.729095:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57230 00000100:00000040:7.0:1496184934.729096:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.729099:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.729105:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.729150:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.730659:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184934.741714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00010000:00020000:7.0:1496184934.752777:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.752834:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.752839:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.752842:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.752847:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.752850:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.752856:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23750us (23839us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.752861:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57230 00000100:00000040:7.0:1496184934.752862:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.752866:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.752872:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.752874:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000020:00000040:7.0:1496184934.752878:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.763734:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092240 00000020:00000040:7.0:1496184934.763737:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.763739:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.763753:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43049 00000100:00000040:7.0:1496184934.763755:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.763758:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c9050 x1568861104092240/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.763765:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092240:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.763770:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.763776:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.763778:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.763788:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.763792:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.763794:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.763795:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.763800:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.763803:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.763804:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.763805:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.763806:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.763808:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.763812:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.763814:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.763816:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.763818:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.763820:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.763823:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.763824:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.763825:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.763829:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092240 00000100:00000040:7.0:1496184934.763833:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.763836:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.763855:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.763857:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c9050 x1568861104092240/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.763863:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092240:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (176us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.763868:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43049 00000100:00000040:7.0:1496184934.763869:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.763873:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.764533:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.764536:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.764537:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.764543:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.764551:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.764557:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.764560:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00000100:00100000:7.0:1496184934.764570:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57231 00000100:00000040:7.0:1496184934.764571:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.764574:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.764580:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.764624:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.766604:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184934.777714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184934.788737:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.788822:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.788827:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.788830:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.788835:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.788838:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.788844:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24263us (24362us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.788849:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57231 00000100:00000040:7.0:1496184934.788850:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.788854:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.788865:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.788867:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00000020:00000040:7.0:1496184934.788871:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:3.0:1496184934.799391:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f850 x1568861104092272 msgsize 520 00000100:00100000:3.0:1496184934.799400:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184934.799401:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184934.799447:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092272 00000020:00000040:0.0:1496184934.799451:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184934.799455:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184934.799481:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6021 00000100:00000040:0.0:1496184934.799484:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184934.799489:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f850 x1568861104092272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184934.799497:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092272:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184934.799518:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184934.799533:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184934.799537:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184934.799541:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:0.0:1496184934.799550:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184934.799553:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184934.799555:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184934.799563:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184934.799568:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184934.799570:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184934.799571:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184934.799572:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184934.799575:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.799581:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184934.799583:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184934.799586:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.799588:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184934.799603:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184934.799606:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184934.799608:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184934.799610:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184934.799617:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092272 00000100:00000040:0.0:1496184934.799624:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.799628:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.799647:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.799650:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f850 x1568861104092272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184934.799664:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092272:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (272us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184934.799670:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6021 00000100:00000040:0.0:1496184934.799672:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184934.799678:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184934.800140:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075850 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184934.800149:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184934.800151:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:7.0:1496184934.800175:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.800176:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.800178:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.800183:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.800189:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.800194:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.800196:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00000100:00100000:7.0:1496184934.800205:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57232 00000100:00000040:7.0:1496184934.800206:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.800209:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.800215:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.800257:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.801969:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184934.813055:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184934.824070:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.824131:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.824136:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.824139:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.824144:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.824146:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.824156:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23941us (24016us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.824162:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57232 00000100:00000040:7.0:1496184934.824163:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.824167:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.824173:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.824175:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00000020:00000040:7.0:1496184934.824179:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:3.0:1496184934.834070:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0050 x1568861104092304 msgsize 520 00000100:00100000:3.0:1496184934.834078:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184934.834079:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184934.834114:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092304 00000020:00000040:0.0:1496184934.834119:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184934.834123:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184934.834144:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6022 00000100:00000040:0.0:1496184934.834147:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184934.834150:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0050 x1568861104092304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184934.834158:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092304:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184934.834169:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184934.834179:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184934.834183:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184934.834186:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:0.0:1496184934.834193:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184934.834195:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184934.834197:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184934.834204:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184934.834207:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184934.834209:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184934.834210:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184934.834211:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184934.834214:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.834218:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184934.834221:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184934.834226:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.834228:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184934.834233:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184934.834236:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184934.834238:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184934.834239:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184934.834246:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092304 00000100:00000040:0.0:1496184934.834253:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.834256:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.834273:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184934.834276:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0050 x1568861104092304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184934.834282:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092304:12345-10.37.248.124@o2ib1:8 Request procesed in 124us (211us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184934.834287:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6022 00000100:00000040:0.0:1496184934.834289:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184934.834293:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.834743:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.834745:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.834747:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.834751:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.834758:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.834763:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.834765:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00000100:00100000:7.0:1496184934.834772:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57233 00000100:00000040:7.0:1496184934.834773:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.834776:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.834782:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.834824:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.836516:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184934.847577:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00000100:00000040:1.0:1496184934.858663:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f050 x1568861104092336 msgsize 520 00000100:00100000:1.0:1496184934.858671:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184934.858681:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00010000:00020000:7.0:1496184934.858683:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00100000:2.0:1496184934.858700:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092336 00000020:00000040:2.0:1496184934.858704:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184934.858706:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:2.0:1496184934.858724:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6023 00000100:00000040:2.0:1496184934.858726:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000020:00000040:7.0:1496184934.858729:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184934.858729:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f050 x1568861104092336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:02000400:7.0:1496184934.858734:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.858737:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184934.858737:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104092336:12345-10.37.248.124@o2ib1:8 00010000:00000040:7.0:1496184934.858743:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000040:2.0:1496184934.858744:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000040:7.0:1496184934.858746:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184934.858751:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:7.0:1496184934.858752:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+10:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23970us (24040us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.858757:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57233 00010000:02000400:2.0:1496184934.858757:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00000100:00000040:7.0:1496184934.858759:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00010000:00080000:2.0:1496184934.858760:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00002000:00100000:7.0:1496184934.858763:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000001:00000040:2.0:1496184934.858766:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184934.858768:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.858769:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00002000:00100000:2.0:1496184934.858769:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00010000:7.0:1496184934.858771:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00000020:00000040:7.0:1496184934.858776:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00002000:00100000:2.0:1496184934.858776:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184934.858780:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184934.858781:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184934.858782:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184934.858783:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184934.858785:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184934.858790:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184934.858791:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184934.858794:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184934.858796:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184934.858799:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184934.858802:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184934.858803:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184934.858805:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184934.858810:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092336 00000100:00000040:2.0:1496184934.858815:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184934.858826:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184934.858840:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184934.858843:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f050 x1568861104092336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184934.858849:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092336:12345-10.37.248.124@o2ib1:8 Request procesed in 113us (186us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184934.858855:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6023 00000100:00000040:2.0:1496184934.858856:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184934.858861:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:4.0:1496184934.859778:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:4.0:1496184934.859786:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:4.0:1496184934.859788:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:7.0:1496184934.859837:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.859842:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.859844:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.859863:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.859871:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.859878:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.859880:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00000100:00100000:7.0:1496184934.859899:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57234 00000100:00000040:7.0:1496184934.859901:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.859903:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.859914:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.859963:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.870180:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184934.881183:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184934.892241:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.892286:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.892290:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.892293:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.892299:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.892301:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.892307:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32393us (32528us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.892312:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57234 00000100:00000040:7.0:1496184934.892314:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.892318:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.892323:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.892326:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00000020:00000040:7.0:1496184934.892329:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.904080:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092368 00000020:00000040:7.0:1496184934.904084:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.904085:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.904100:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43050 00000100:00000040:7.0:1496184934.904101:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.904104:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8c50 x1568861104092368/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.904112:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092368:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.904118:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.904123:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.904126:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.904129:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.904133:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.904135:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.904137:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.904141:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.904144:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.904146:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.904146:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.904147:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.904150:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.904154:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.904156:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.904158:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.904160:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.904163:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.904166:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.904168:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.904169:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.904172:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092368 00000100:00000040:7.0:1496184934.904177:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.904179:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.904191:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.904194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8c50 x1568861104092368/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.904200:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092368:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (167us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.904204:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43050 00000100:00000040:7.0:1496184934.904206:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.904209:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.904728:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.904731:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.904733:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.904738:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.904746:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.904752:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.904755:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00000100:00100000:7.0:1496184934.904765:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57235 00000100:00000040:7.0:1496184934.904766:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.904768:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.904775:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.904817:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.906446:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184934.906470:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184934.906514:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.906559:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.906564:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.906567:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.906576:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.906579:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.906585:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1809us (1900us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.906589:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57235 00000100:00000040:7.0:1496184934.906591:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.906595:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.906600:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.906603:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00000020:00000040:7.0:1496184934.906606:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.935291:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092400 00000020:00000040:7.0:1496184934.935294:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.935295:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.935310:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43051 00000100:00000040:7.0:1496184934.935311:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.935314:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8050 x1568861104092400/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.935321:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092400:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.935327:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.935332:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.935335:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.935337:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.935341:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.935343:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.935344:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.935349:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.935352:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.935353:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.935354:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.935355:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.935357:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.935361:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.935362:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.935364:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.935366:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.935369:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.935371:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.935372:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.935374:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.935377:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092400 00000100:00000040:7.0:1496184934.935381:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.935384:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.935396:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.935398:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8050 x1568861104092400/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.935404:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092400:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (157us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.935409:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43051 00000100:00000040:7.0:1496184934.935410:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.935413:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.935950:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.935953:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.935955:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.935960:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.935968:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.935975:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.935977:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00000100:00100000:7.0:1496184934.935987:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57236 00000100:00000040:7.0:1496184934.935989:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.935991:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.935997:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.936042:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.937717:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184934.937742:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184934.937787:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.937832:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.937836:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.937840:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.937844:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.937847:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.937853:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1855us (1947us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.937858:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57236 00000100:00000040:7.0:1496184934.937859:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.937863:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.937869:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.937872:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00000020:00000040:7.0:1496184934.937875:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.966170:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092432 00000020:00000040:7.0:1496184934.966174:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.966175:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.966190:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43052 00000100:00000040:7.0:1496184934.966191:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.966194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7450 x1568861104092432/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.966201:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092432:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.966207:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.966212:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.966215:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.966217:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.966221:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.966223:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.966225:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.966229:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.966232:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.966233:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.966234:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.966235:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.966237:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.966241:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.966242:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.966245:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.966247:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.966249:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.966252:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.966253:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.966254:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.966258:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092432 00000100:00000040:7.0:1496184934.966262:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.966265:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.966277:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.966279:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7450 x1568861104092432/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.966285:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092432:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.966290:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43052 00000100:00000040:7.0:1496184934.966291:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.966295:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.966791:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.966795:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.966796:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.966802:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.966810:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.966817:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.966819:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00000100:00100000:7.0:1496184934.966829:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57237 00000100:00000040:7.0:1496184934.966830:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.966833:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.966839:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.966883:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.968586:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184934.968612:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184934.968654:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184934.968707:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184934.968712:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184934.968715:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184934.968720:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184934.968722:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184934.968728:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1889us (1980us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184934.968733:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57237 00000100:00000040:7.0:1496184934.968735:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184934.968738:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184934.968744:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.968746:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00000020:00000040:7.0:1496184934.968750:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.996695:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092464 00000020:00000040:7.0:1496184934.996698:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.996700:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184934.996714:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43053 00000100:00000040:7.0:1496184934.996716:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.996719:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7c50 x1568861104092464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184954 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.996726:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092464:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184934.996732:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184934.996737:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184934.996740:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184934.996742:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184934 last 1496184933 00000001:00000040:7.0:1496184934.996746:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184934.996748:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184934.996749:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184934.996754:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184934.996757:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.996758:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.996759:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184934.996759:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.996761:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.996765:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.996767:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184934.996769:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.996771:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184934.996774:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184934.996776:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184934.996777:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184934.996779:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184934.996782:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092464 00000100:00000040:7.0:1496184934.996787:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.996789:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.996801:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184934.996804:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7c50 x1568861104092464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184954 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184934.996810:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092464:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184934.996814:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43053 00000100:00000040:7.0:1496184934.996816:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184934.996819:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184934.997379:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184934.997382:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184934.997383:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184934.997389:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184934.997396:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184934.997403:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184934.997406:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00000100:00100000:7.0:1496184934.997415:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57238 00000100:00000040:7.0:1496184934.997417:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184934.997419:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/0 e 0 to 0 dl 1496184977 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184934.997426:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184934.997468:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184934.998887:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184935.010057:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184935.021129:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.021174:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.021179:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.021182:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.021187:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.021190:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.021196:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23770us (23863us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.021201:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57238 00000100:00000040:7.0:1496184935.021202:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.021206:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:722/0 lens 608/448 e 0 to 0 dl 1496184977 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.021212:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.021214:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00000020:00000040:7.0:1496184935.021218:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.031964:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092496 00000020:00000040:7.0:1496184935.031967:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.031968:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.031983:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43054 00000100:00000040:7.0:1496184935.031985:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.031988:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6c50 x1568861104092496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.031995:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092496:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.032001:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.032006:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.032008:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.032011:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.032015:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.032017:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.032018:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.032025:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.032028:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.032029:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.032030:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.032031:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.032033:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.032037:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.032039:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.032041:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.032043:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.032045:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.032048:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.032049:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.032051:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.032054:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092496 00000100:00000040:7.0:1496184935.032058:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.032061:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.032073:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.032075:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6c50 x1568861104092496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.032082:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092496:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.032086:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43054 00000100:00000040:7.0:1496184935.032088:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.032091:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.032666:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.032670:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.032671:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.032676:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.032684:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.032691:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.032693:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00000100:00100000:7.0:1496184935.032703:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57239 00000100:00000040:7.0:1496184935.032705:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.032707:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.032714:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.032756:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.034301:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184935.045358:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184935.056461:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184935.056474:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8fc50 x1568861104092528 msgsize 520 00000100:00100000:1.0:1496184935.056483:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.056487:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184935.056507:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.056511:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.056515:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.056520:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.056522:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.056528:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23814us (23906us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.056533:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57239 00000100:00000040:7.0:1496184935.056535:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:2.0:1496184935.056536:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092528 00002000:00100000:7.0:1496184935.056539:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184935.056541:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00100000:7.0:1496184935.056544:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00000020:00000040:2.0:1496184935.056545:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00010000:7.0:1496184935.056547:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00000020:00000040:7.0:1496184935.056551:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.056574:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6024 00000100:00000040:2.0:1496184935.056576:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.056581:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8fc50 x1568861104092528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184935.056596:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092528:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.056608:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.056621:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.056625:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.056629:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.056638:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.056642:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.056644:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.056653:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.056657:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.056659:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.056660:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.056661:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.056664:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.056670:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.056673:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.056678:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.056680:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.056686:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.056689:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.056691:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.056693:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.056702:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092528 00000100:00000040:2.0:1496184935.056709:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.056713:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.056741:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.056744:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8fc50 x1568861104092528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.056750:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092528:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (275us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.056755:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6024 00000100:00000040:2.0:1496184935.056757:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.056762:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.057665:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.057668:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.057670:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.057676:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.057684:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.057691:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.057694:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00000100:00100000:7.0:1496184935.057703:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57240 00000100:00000040:7.0:1496184935.057705:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.057707:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.057714:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.057755:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.070166:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184935.081178:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184935.092231:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.092276:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.092281:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.092284:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.092289:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.092292:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.092298:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34584us (34679us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.092303:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57240 00000100:00000040:7.0:1496184935.092305:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.092309:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.092314:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.092317:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00000020:00000040:7.0:1496184935.092320:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:1.0:1496184935.102899:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ec50 x1568861104092560 msgsize 520 00000100:00100000:1.0:1496184935.102907:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.102909:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184935.102926:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092560 00000020:00000040:2.0:1496184935.102930:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184935.102933:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.102953:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6025 00000100:00000040:2.0:1496184935.102956:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.102960:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ec50 x1568861104092560/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184935.102968:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092560:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.102978:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.102987:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.102990:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.102996:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.103001:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.103004:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.103006:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.103012:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.103015:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.103017:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.103018:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.103019:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.103021:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.103026:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.103028:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.103031:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.103033:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.103037:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.103040:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.103042:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.103043:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.103049:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092560 00000100:00000040:2.0:1496184935.103055:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.103059:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.103081:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.103083:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ec50 x1568861104092560/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.103089:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092560:12345-10.37.248.124@o2ib1:8 Request procesed in 122us (189us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.103097:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6025 00000100:00000040:2.0:1496184935.103099:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.103103:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.103571:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.103574:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.103576:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.103583:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.103590:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.103597:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.103599:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00000100:00100000:7.0:1496184935.103609:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57241 00000100:00000040:7.0:1496184935.103611:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.103613:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.103620:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.103662:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.105149:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184935.105164:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184935.105211:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.105260:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.105264:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.105268:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.105272:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.105275:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.105281:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1661us (1755us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.105286:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57241 00000100:00000040:7.0:1496184935.105288:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.105291:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.105297:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.105300:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00000020:00000040:7.0:1496184935.105303:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.133745:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092592 00000020:00000040:7.0:1496184935.133748:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.133750:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.133764:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43055 00000100:00000040:7.0:1496184935.133766:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.133769:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7050 x1568861104092592/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.133776:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092592:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.133782:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.133787:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.133790:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.133793:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.133797:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.133799:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.133800:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.133805:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.133808:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.133809:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.133810:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.133811:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.133813:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.133818:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.133819:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.133822:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.133823:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.133826:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.133829:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.133830:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.133832:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.133835:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092592 00000100:00000040:7.0:1496184935.133839:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.133842:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.133859:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.133861:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7050 x1568861104092592/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.133867:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092592:12345-10.37.248.124@o2ib1:8 Request procesed in 92us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.133872:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43055 00000100:00000040:7.0:1496184935.133873:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.133877:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.134420:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.134423:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.134424:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.134430:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.134438:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.134445:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.134447:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00000100:00100000:7.0:1496184935.134457:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57242 00000100:00000040:7.0:1496184935.134458:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.134461:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.134467:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.134509:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.136040:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184935.136050:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184935.136094:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.136140:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.136144:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.136147:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.136152:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.136155:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.136165:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1698us (1789us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.136170:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57242 00000100:00000040:7.0:1496184935.136172:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.136175:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.136181:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.136184:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00000020:00000040:7.0:1496184935.136187:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.164554:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092624 00000020:00000040:7.0:1496184935.164557:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.164559:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.164573:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43056 00000100:00000040:7.0:1496184935.164575:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.164578:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7850 x1568861104092624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.164585:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092624:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.164591:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.164596:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.164599:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.164601:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.164605:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.164607:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.164608:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.164613:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.164616:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.164617:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.164618:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.164619:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.164621:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.164625:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.164627:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.164629:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.164631:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.164633:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.164636:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.164637:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.164638:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.164642:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092624 00000100:00000040:7.0:1496184935.164646:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.164649:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.164661:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.164663:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7850 x1568861104092624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.164670:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092624:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.164674:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43056 00000100:00000040:7.0:1496184935.164675:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.164679:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.165150:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.165153:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.165155:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.165161:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.165169:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.165175:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.165178:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00000100:00100000:7.0:1496184935.165187:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57243 00000100:00000040:7.0:1496184935.165189:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.165191:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.165198:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.165239:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.166607:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184935.166616:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184935.166660:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.166705:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.166710:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.166713:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.166718:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.166720:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.166726:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1529us (1621us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.166731:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57243 00000100:00000040:7.0:1496184935.166733:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.166737:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.166742:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.166745:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00000020:00000040:7.0:1496184935.166748:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.194257:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092656 00000020:00000040:7.0:1496184935.194260:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.194262:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.194276:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43057 00000100:00000040:7.0:1496184935.194278:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.194280:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6850 x1568861104092656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.194288:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092656:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.194293:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.194299:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.194301:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.194304:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.194308:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.194309:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.194311:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.194315:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.194318:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.194319:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.194320:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.194321:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.194323:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.194327:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.194328:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.194331:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.194333:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.194335:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.194338:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.194339:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.194341:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.194344:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092656 00000100:00000040:7.0:1496184935.194348:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.194351:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.194364:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.194366:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6850 x1568861104092656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.194372:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092656:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.194377:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43057 00000100:00000040:7.0:1496184935.194378:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.194382:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.194916:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.194919:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.194921:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.194934:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.194942:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.194949:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.194951:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00000100:00100000:7.0:1496184935.194961:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57244 00000100:00000040:7.0:1496184935.194962:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.194965:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.194971:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.195018:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.196485:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184935.207625:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184935.218693:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.218738:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.218743:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.218745:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.218751:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.218753:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.218759:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23788us (23889us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.218764:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57244 00000100:00000040:7.0:1496184935.218766:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.218769:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.218775:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.218778:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00000020:00000040:7.0:1496184935.218781:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.229210:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092688 00000020:00000040:7.0:1496184935.229221:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.229223:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.229237:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43058 00000100:00000040:7.0:1496184935.229239:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.229242:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6050 x1568861104092688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.229249:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092688:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.229255:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.229260:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.229263:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.229266:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.229269:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.229271:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.229272:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.229277:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.229280:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.229281:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.229282:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.229283:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.229285:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.229289:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.229290:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.229293:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.229295:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.229297:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.229300:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.229301:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.229302:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.229306:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092688 00000100:00000040:7.0:1496184935.229310:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.229313:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.229324:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.229327:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6050 x1568861104092688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.229333:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092688:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (171us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.229337:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43058 00000100:00000040:7.0:1496184935.229339:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.229342:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.229905:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.229908:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.229910:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.229915:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.229923:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.229930:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.229932:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00000100:00100000:7.0:1496184935.229942:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57245 00000100:00000040:7.0:1496184935.229944:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.229946:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.229952:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.229995:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.231561:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184935.242610:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184935.253702:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184935.253728:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e050 x1568861104092720 msgsize 520 00000100:00100000:1.0:1496184935.253737:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.253739:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184935.253760:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.253765:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.253768:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.253773:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.253775:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.253781:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23829us (23929us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.253786:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57245 00000100:00000040:7.0:1496184935.253788:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:2.0:1496184935.253789:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092720 00002000:00100000:7.0:1496184935.253792:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184935.253794:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00100000:7.0:1496184935.253797:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00000020:00000040:2.0:1496184935.253798:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00010000:7.0:1496184935.253800:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00000020:00000040:7.0:1496184935.253803:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.253827:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6026 00000100:00000040:2.0:1496184935.253829:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.253834:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e050 x1568861104092720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184935.253843:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092720:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.253863:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.253875:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.253880:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.253884:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.253899:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.253902:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.253905:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.253913:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.253917:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.253920:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.253921:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.253922:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.253925:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.253930:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.253933:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.253939:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.253941:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.253946:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.253949:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.253952:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.253954:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.253962:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092720 00000100:00000040:2.0:1496184935.253970:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.253974:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.254002:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.254005:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e050 x1568861104092720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.254011:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092720:12345-10.37.248.124@o2ib1:8 Request procesed in 169us (282us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.254016:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6026 00000100:00000040:2.0:1496184935.254018:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.254024:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.254912:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.254915:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.254917:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.254922:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.254930:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.254937:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.254939:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00000100:00100000:7.0:1496184935.254950:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57246 00000100:00000040:7.0:1496184935.254952:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.254954:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.254961:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.255004:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.256652:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184935.267652:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184935.278690:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.278739:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.278744:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.278747:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.278752:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.278754:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.278760:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23799us (23917us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.278765:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57246 00000100:00000040:7.0:1496184935.278767:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.278771:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.278776:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.278779:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00000020:00000040:7.0:1496184935.278782:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.294912:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092752 00000020:00000040:7.0:1496184935.294915:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.294916:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.294931:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43059 00000100:00000040:7.0:1496184935.294932:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.294935:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5c50 x1568861104092752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.294942:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092752:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.294948:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.294954:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.294957:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.294959:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.294963:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.294965:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.294967:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.294972:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.294975:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.294976:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.294977:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.294978:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.294980:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.294984:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.294986:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.294988:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.294990:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.294993:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.294996:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.294998:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.294999:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.295002:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092752 00000100:00000040:7.0:1496184935.295006:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.295009:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.295021:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.295023:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5c50 x1568861104092752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.295029:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092752:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (177us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.295034:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43059 00000100:00000040:7.0:1496184935.295035:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.295038:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.295598:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.295601:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.295602:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.295608:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.295615:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.295622:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.295624:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00000100:00100000:7.0:1496184935.295634:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57247 00000100:00000040:7.0:1496184935.295636:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.295638:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.295645:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.295688:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.297315:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184935.297338:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00010000:00020000:7.0:1496184935.297382:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.297427:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.297432:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.297435:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.297440:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.297442:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.297452:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1807us (1904us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.297457:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57247 00000100:00000040:7.0:1496184935.297459:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.297463:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.297468:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.297471:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00000020:00000040:7.0:1496184935.297474:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.326525:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092784 00000020:00000040:7.0:1496184935.326528:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.326530:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.326544:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43060 00000100:00000040:7.0:1496184935.326546:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.326549:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5850 x1568861104092784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.326556:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092784:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.326562:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.326567:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.326570:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.326572:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.326576:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.326578:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.326579:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.326584:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.326587:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.326588:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.326589:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.326590:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.326592:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.326596:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.326597:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.326600:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.326602:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.326604:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.326607:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.326608:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.326610:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.326613:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092784 00000100:00000040:7.0:1496184935.326617:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.326620:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.326632:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.326634:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5850 x1568861104092784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.326640:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092784:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.326645:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43060 00000100:00000040:7.0:1496184935.326646:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.326649:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.327201:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.327204:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.327206:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.327212:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.327219:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.327226:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.327229:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00000100:00100000:7.0:1496184935.327238:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57248 00000100:00000040:7.0:1496184935.327240:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.327243:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.327249:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.327293:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.328809:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184935.328836:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00010000:00020000:7.0:1496184935.328878:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.328936:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.328941:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.328944:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.328949:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.328952:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.328958:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1709us (1801us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.328963:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57248 00000100:00000040:7.0:1496184935.328964:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.328968:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.328974:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.328977:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00000020:00000040:7.0:1496184935.328980:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.357257:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092816 00000020:00000040:7.0:1496184935.357260:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.357262:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.357276:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43061 00000100:00000040:7.0:1496184935.357278:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.357281:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6450 x1568861104092816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.357288:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092816:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.357294:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.357299:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.357302:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.357304:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.357308:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.357310:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.357311:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.357316:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.357319:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.357320:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.357321:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.357322:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.357324:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.357328:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.357329:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.357332:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.357334:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.357336:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.357339:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.357341:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.357342:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.357345:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092816 00000100:00000040:7.0:1496184935.357349:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.357352:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.357364:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.357366:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6450 x1568861104092816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.357372:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092816:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.357377:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43061 00000100:00000040:7.0:1496184935.357378:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.357382:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.357942:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.357945:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.357947:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.357952:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.357960:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.357967:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.357969:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00000100:00100000:7.0:1496184935.357979:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57249 00000100:00000040:7.0:1496184935.357980:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.357983:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.357989:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.358032:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.359560:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184935.359576:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184935.359630:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.359675:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.359680:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.359683:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.359688:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.359691:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.359697:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1707us (1810us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.359702:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57249 00000100:00000040:7.0:1496184935.359703:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.359707:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.359713:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.359715:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00000020:00000040:7.0:1496184935.359718:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.387499:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092848 00000020:00000040:7.0:1496184935.387502:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.387503:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.387517:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43062 00000100:00000040:7.0:1496184935.387519:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.387522:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8450 x1568861104092848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.387529:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092848:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.387535:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.387540:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.387543:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.387546:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.387549:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.387551:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.387552:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.387557:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.387560:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.387561:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.387562:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.387563:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.387565:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.387569:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.387570:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.387572:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.387574:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.387577:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.387579:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.387581:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.387582:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.387586:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092848 00000100:00000040:7.0:1496184935.387590:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.387592:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.387604:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.387607:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8450 x1568861104092848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.387613:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092848:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.387617:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43062 00000100:00000040:7.0:1496184935.387619:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.387622:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.388125:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.388128:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.388130:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.388135:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.388143:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.388149:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.388152:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00000100:00100000:7.0:1496184935.388161:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57250 00000100:00000040:7.0:1496184935.388163:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.388165:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.388172:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.388214:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.389912:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184935.389935:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184935.389978:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.390024:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.390028:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.390031:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.390036:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.390039:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.390045:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1873us (1965us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.390050:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57250 00000100:00000040:7.0:1496184935.390052:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.390055:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.390061:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.390063:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00000020:00000040:7.0:1496184935.390067:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.417688:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092880 00000020:00000040:7.0:1496184935.417691:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.417692:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.417706:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43063 00000100:00000040:7.0:1496184935.417708:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.417711:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8850 x1568861104092880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.417718:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092880:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.417724:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.417729:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.417732:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.417734:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.417738:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.417740:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.417741:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.417746:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.417749:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.417750:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.417751:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.417752:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.417754:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.417758:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.417760:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.417762:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.417764:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.417766:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.417768:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.417770:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.417771:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.417775:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092880 00000100:00000040:7.0:1496184935.417779:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.417781:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.417793:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.417796:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8850 x1568861104092880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.417802:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092880:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.417806:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43063 00000100:00000040:7.0:1496184935.417808:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.417811:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.418411:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.418414:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.418416:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.418421:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.418429:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.418436:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.418438:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00000100:00100000:7.0:1496184935.418448:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57251 00000100:00000040:7.0:1496184935.418449:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.418452:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.418458:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.418501:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.420057:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184935.431199:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184935.442262:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.442320:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.442325:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.442328:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.442333:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.442336:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.442342:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23883us (23977us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.442347:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57251 00000100:00000040:7.0:1496184935.442348:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.442352:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.442362:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.442365:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00000020:00000040:7.0:1496184935.442368:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.453369:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092912 00000020:00000040:7.0:1496184935.453372:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.453374:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.453388:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43064 00000100:00000040:7.0:1496184935.453390:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.453393:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5050 x1568861104092912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.453400:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092912:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.453406:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.453411:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.453414:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.453416:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.453420:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.453422:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.453423:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.453427:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.453430:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.453431:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.453432:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.453433:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.453435:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.453439:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.453441:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.453443:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.453445:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.453448:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.453450:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.453452:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.453453:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.453457:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092912 00000100:00000040:7.0:1496184935.453461:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.453463:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.453475:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.453478:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5050 x1568861104092912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.453484:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092912:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.453488:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43064 00000100:00000040:7.0:1496184935.453489:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.453493:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.454106:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.454109:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.454110:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.454116:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.454123:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.454130:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.454132:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00000100:00100000:7.0:1496184935.454142:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57252 00000100:00000040:7.0:1496184935.454144:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.454146:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.454153:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.454195:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.455751:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184935.466821:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184935.477872:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.477940:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.477945:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.477948:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.477953:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.477956:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.477962:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23809us (23901us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.477967:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57252 00000100:00000040:7.0:1496184935.477968:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.477972:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.477978:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.477980:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00000020:00000040:7.0:1496184935.477983:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.488742:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092944 00000020:00000040:7.0:1496184935.488745:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.488747:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.488761:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43065 00000100:00000040:7.0:1496184935.488763:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.488766:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5450 x1568861104092944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.488773:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092944:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.488779:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.488784:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.488787:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.488789:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.488793:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.488795:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.488796:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.488801:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.488804:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.488805:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.488806:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.488807:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.488809:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.488813:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.488814:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.488817:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.488819:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.488821:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.488823:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.488825:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.488826:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.488830:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092944 00000100:00000040:7.0:1496184935.488834:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.488837:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.488849:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.488851:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5450 x1568861104092944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.488857:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092944:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.488862:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43065 00000100:00000040:7.0:1496184935.488863:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.488866:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.489473:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.489476:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.489477:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.489483:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.489491:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.489498:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.489500:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000100:00100000:7.0:1496184935.489510:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57253 00000100:00000040:7.0:1496184935.489512:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.489514:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.489521:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.489565:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.491308:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184935.502357:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184935.513429:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.513486:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.513491:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.513494:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.513499:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.513502:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.513508:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23987us (24079us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.513513:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57253 00000100:00000040:7.0:1496184935.513515:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.513518:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.513524:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.513526:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000020:00000040:7.0:1496184935.513530:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.524607:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104092976 00000020:00000040:7.0:1496184935.524610:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.524612:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.524626:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43066 00000100:00000040:7.0:1496184935.524627:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.524630:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4850 x1568861104092976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.524637:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092976:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.524643:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.524648:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.524651:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.524654:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.524658:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.524659:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.524661:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.524665:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.524668:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.524670:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.524670:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.524671:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.524673:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.524677:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.524679:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.524681:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.524683:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.524685:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.524688:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.524689:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.524691:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.524694:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104092976 00000100:00000040:7.0:1496184935.524698:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.524701:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.524713:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.524715:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4850 x1568861104092976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.524721:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104092976:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.524725:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43066 00000100:00000040:7.0:1496184935.524727:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.524730:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.525342:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.525345:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.525347:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.525352:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.525360:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.525367:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.525369:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000100:00100000:7.0:1496184935.525379:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57254 00000100:00000040:7.0:1496184935.525381:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.525383:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.525390:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.525432:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.527158:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184935.538204:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184935.549249:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.549307:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.549312:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.549315:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.549320:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.549323:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.549329:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23939us (24037us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.549334:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57254 00000100:00000040:7.0:1496184935.549335:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.549339:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.549345:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.549347:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000020:00000040:7.0:1496184935.549351:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.559965:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093008 00000020:00000040:7.0:1496184935.559969:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.559972:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.559993:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43067 00000100:00000040:7.0:1496184935.559996:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.560000:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4c50 x1568861104093008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.560025:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093008:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.560030:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.560036:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.560039:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.560041:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.560045:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.560047:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.560048:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.560053:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.560055:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.560057:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.560058:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.560058:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.560060:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.560065:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.560066:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.560068:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.560070:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.560073:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.560075:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.560077:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.560078:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.560081:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093008 00000100:00000040:7.0:1496184935.560085:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.560088:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.560101:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.560103:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4c50 x1568861104093008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.560109:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093008:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (220us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.560113:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43067 00000100:00000040:7.0:1496184935.560115:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.560118:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.560760:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.560763:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.560764:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.560770:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.560777:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.560784:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.560786:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000100:00100000:7.0:1496184935.560796:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57255 00000100:00000040:7.0:1496184935.560798:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.560800:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.560807:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.560850:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.562455:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184935.573509:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00010000:00020000:7.0:1496184935.584570:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.584616:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.584620:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.584623:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.584628:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.584631:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.584637:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23830us (23931us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.584642:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57255 00000100:00000040:7.0:1496184935.584643:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.584647:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.584653:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.584655:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000020:00000040:7.0:1496184935.584658:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.594958:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093040 00000020:00000040:7.0:1496184935.594961:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.594963:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.594977:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43068 00000100:00000040:7.0:1496184935.594979:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.594981:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4050 x1568861104093040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.594989:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093040:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.594994:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.595000:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.595002:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.595005:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.595009:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.595010:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.595012:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.595016:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.595019:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.595020:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.595021:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.595022:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.595024:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.595028:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.595029:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.595032:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.595033:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.595036:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.595038:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.595040:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.595041:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.595045:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093040 00000100:00000040:7.0:1496184935.595049:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.595051:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.595063:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.595066:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4050 x1568861104093040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.595072:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093040:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.595076:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43068 00000100:00000040:7.0:1496184935.595078:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.595081:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.595697:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.595700:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.595702:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.595708:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.595715:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.595722:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.595725:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00000100:00100000:7.0:1496184935.595734:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57256 00000100:00000040:7.0:1496184935.595736:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.595738:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.595745:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.595788:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.597594:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184935.608599:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184935.619652:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.619711:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.619715:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.619719:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.619724:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.619726:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.619732:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23987us (24083us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.619737:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57256 00000100:00000040:7.0:1496184935.619739:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.619742:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.619748:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.619750:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00000020:00000040:7.0:1496184935.619754:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.630437:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093072 00000020:00000040:7.0:1496184935.630440:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.630442:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.630456:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43069 00000100:00000040:7.0:1496184935.630458:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.630461:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4450 x1568861104093072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.630468:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093072:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.630474:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.630479:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.630481:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.630484:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.630488:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.630490:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.630491:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.630496:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.630499:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.630500:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.630501:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.630502:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.630504:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.630507:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.630509:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.630511:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.630513:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.630516:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.630518:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.630520:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.630521:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.630524:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093072 00000100:00000040:7.0:1496184935.630529:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.630531:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.630543:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.630546:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4450 x1568861104093072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.630552:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093072:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.630556:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43069 00000100:00000040:7.0:1496184935.630557:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.630561:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.631124:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.631127:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.631128:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.631133:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.631141:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.631148:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.631150:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00000100:00100000:7.0:1496184935.631160:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57257 00000100:00000040:7.0:1496184935.631162:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.631164:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.631170:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.631213:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.632818:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184935.643850:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184935.654942:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184935.654962:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f450 x1568861104093104 msgsize 520 00000100:00100000:1.0:1496184935.654971:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.654972:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184935.654993:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093104 00000020:00000040:2.0:1496184935.654998:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.654999:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.655002:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:02000400:7.0:1496184935.655004:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.655007:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.655013:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.655015:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.655021:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23850us (23942us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.655026:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57257 00000100:00000040:7.0:1496184935.655028:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00100000:2.0:1496184935.655030:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6027 00002000:00100000:7.0:1496184935.655032:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:2.0:1496184935.655032:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184935.655038:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00000100:00000040:2.0:1496184935.655038:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f450 x1568861104093104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00010000:7.0:1496184935.655040:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00000020:00000040:7.0:1496184935.655044:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.655047:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093104:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.655059:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.655071:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.655076:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.655082:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.655091:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.655102:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.655104:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.655112:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.655117:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.655119:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.655120:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.655121:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.655124:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.655130:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.655133:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.655136:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.655138:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.655143:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.655146:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.655149:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.655150:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.655159:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093104 00000100:00000040:2.0:1496184935.655166:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.655170:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.655190:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.655193:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f450 x1568861104093104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.655199:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093104:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (237us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.655207:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6027 00000100:00000040:2.0:1496184935.655209:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.655224:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.656160:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.656163:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.656165:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.656170:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.656178:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.656185:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.656187:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00000100:00100000:7.0:1496184935.656197:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57258 00000100:00000040:7.0:1496184935.656198:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.656201:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.656207:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.656250:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.657973:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184935.668981:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184935.680010:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.680067:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.680072:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.680075:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.680080:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.680083:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.680089:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23882us (23973us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.680094:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57258 00000100:00000040:7.0:1496184935.680096:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.680100:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.680105:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.680108:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00000020:00000040:7.0:1496184935.680111:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.693488:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093136 00000020:00000040:7.0:1496184935.693491:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.693493:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.693507:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43070 00000100:00000040:7.0:1496184935.693509:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.693512:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3050 x1568861104093136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.693519:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093136:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.693525:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.693530:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.693533:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.693536:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.693540:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.693542:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.693543:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.693548:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.693551:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.693552:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.693553:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.693554:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.693557:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.693561:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.693562:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.693565:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.693567:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.693569:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.693572:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.693574:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.693575:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.693578:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093136 00000100:00000040:7.0:1496184935.693582:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.693585:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.693597:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.693600:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3050 x1568861104093136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.693606:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093136:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.693610:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43070 00000100:00000040:7.0:1496184935.693612:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.693615:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.694170:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.694173:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.694175:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.694180:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.694188:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.694195:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.694197:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00000100:00100000:7.0:1496184935.694207:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57259 00000100:00000040:7.0:1496184935.694209:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.694211:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.694217:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.694261:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.695625:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184935.706631:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184935.717664:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.717721:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.717726:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.717729:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.717734:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.717736:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.717742:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23524us (23616us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.717747:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57259 00000100:00000040:7.0:1496184935.717749:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.717752:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.717763:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.717765:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00000020:00000040:7.0:1496184935.717768:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.728361:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093168 00000020:00000040:7.0:1496184935.728364:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.728366:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.728381:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43071 00000100:00000040:7.0:1496184935.728382:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.728385:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3c50 x1568861104093168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.728392:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093168:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.728398:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.728403:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.728406:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.728408:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.728412:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.728414:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.728415:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.728420:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.728423:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.728424:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.728425:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.728426:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.728428:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.728432:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.728433:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.728436:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.728438:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.728440:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.728443:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.728444:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.728445:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.728449:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093168 00000100:00000040:7.0:1496184935.728453:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.728456:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.728468:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.728471:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3c50 x1568861104093168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.728477:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093168:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.728481:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43071 00000100:00000040:7.0:1496184935.728483:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.728486:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.729114:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.729117:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.729119:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.729125:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.729133:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.729139:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.729142:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000100:00100000:7.0:1496184935.729151:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57260 00000100:00000040:7.0:1496184935.729153:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.729155:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.729162:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.729205:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.730807:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184935.741847:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00010000:00020000:7.0:1496184935.752926:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.752971:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.752976:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.752979:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.752984:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.752987:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.752993:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23830us (23922us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.752998:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57260 00000100:00000040:7.0:1496184935.752999:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.753003:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.753009:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.753011:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000020:00000040:7.0:1496184935.753014:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.763908:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093200 00000020:00000040:7.0:1496184935.763912:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.763913:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.763927:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43072 00000100:00000040:7.0:1496184935.763929:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.763932:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3850 x1568861104093200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.763939:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093200:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.763945:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.763950:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.763953:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.763956:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.763960:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.763961:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.763963:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.763967:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.763970:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.763971:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.763972:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.763973:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.763975:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.763979:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.763981:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.763983:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.763985:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.763988:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.763990:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.763992:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.763993:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.763996:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093200 00000100:00000040:7.0:1496184935.764000:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.764003:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.764015:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.764018:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3850 x1568861104093200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.764024:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093200:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (171us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.764028:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43072 00000100:00000040:7.0:1496184935.764030:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.764033:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.764655:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.764658:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.764660:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.764665:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.764673:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.764680:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.764682:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000100:00100000:7.0:1496184935.764692:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57261 00000100:00000040:7.0:1496184935.764694:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.764696:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.764703:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.764746:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.766374:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184935.777409:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184935.788465:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.788523:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.788528:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.788531:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.788536:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.788539:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.788545:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23842us (23940us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.788550:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57261 00000100:00000040:7.0:1496184935.788551:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.788555:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.788560:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.788563:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000020:00000040:7.0:1496184935.788566:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.799294:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093232 00000020:00000040:7.0:1496184935.799297:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.799298:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.799313:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43073 00000100:00000040:7.0:1496184935.799314:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.799317:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3450 x1568861104093232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.799325:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093232:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.799330:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.799335:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.799338:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.799341:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.799344:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.799346:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.799348:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.799352:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.799355:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.799356:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.799357:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.799358:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.799360:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.799364:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.799366:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.799368:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.799370:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.799372:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.799374:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.799376:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.799377:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.799381:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093232 00000100:00000040:7.0:1496184935.799385:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.799388:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.799400:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.799402:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3450 x1568861104093232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.799408:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093232:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.799412:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43073 00000100:00000040:7.0:1496184935.799414:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.799417:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.800009:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.800012:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.800013:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.800018:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.800026:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.800033:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.800035:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:7.0:1496184935.800045:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57262 00000100:00000040:7.0:1496184935.800047:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.800049:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.800056:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.800099:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.801845:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:3.0:1496184935.812857:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184935.823925:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.823970:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.823974:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.823977:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.823982:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.823985:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.823991:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23935us (24032us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.823996:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57262 00000100:00000040:7.0:1496184935.823998:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.824002:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.824007:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.824010:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000020:00000040:7.0:1496184935.824013:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.834280:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093264 00000020:00000040:7.0:1496184935.834283:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.834285:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.834299:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43074 00000100:00000040:7.0:1496184935.834301:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.834304:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2450 x1568861104093264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.834311:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093264:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.834317:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.834322:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.834324:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.834327:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.834331:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.834332:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.834334:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.834338:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.834341:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.834343:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.834343:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.834344:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.834346:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.834351:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.834352:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.834354:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.834356:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.834359:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.834361:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.834363:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.834364:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.834368:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093264 00000100:00000040:7.0:1496184935.834372:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.834374:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.834386:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.834389:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2450 x1568861104093264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.834395:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093264:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.834399:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43074 00000100:00000040:7.0:1496184935.834401:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.834404:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.835000:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.835003:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.835005:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.835011:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.835019:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.835025:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.835027:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:7.0:1496184935.835037:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57263 00000100:00000040:7.0:1496184935.835039:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.835041:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.835048:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.835091:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.836592:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184935.847633:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184935.858711:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184935.858735:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e850 x1568861104093296 msgsize 520 00000100:00100000:1.0:1496184935.858744:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.858745:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184935.858768:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.858773:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.858776:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.858781:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.858784:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.858790:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23742us (23843us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.858795:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57263 00000100:00000040:7.0:1496184935.858796:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.858800:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184935.858802:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093296 00002000:00100000:7.0:1496184935.858806:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00000020:00000040:2.0:1496184935.858807:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00010000:7.0:1496184935.858808:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000020:00000040:2.0:1496184935.858811:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:00000040:7.0:1496184935.858812:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.858839:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6028 00000100:00000040:2.0:1496184935.858842:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.858847:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e850 x1568861104093296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184935.858863:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093296:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.858875:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.858887:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.858902:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.858905:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.858914:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.858917:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.858920:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.858928:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.858933:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.858935:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.858936:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.858937:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.858940:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.858946:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.858949:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.858954:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.858957:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.858962:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.858965:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.858967:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.858969:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.858977:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093296 00000100:00000040:2.0:1496184935.858985:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.858989:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.859017:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.859020:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e850 x1568861104093296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.859027:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093296:12345-10.37.248.124@o2ib1:8 Request procesed in 164us (291us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.859031:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6028 00000100:00000040:2.0:1496184935.859034:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.859039:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.859908:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.859911:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.859913:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.859919:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.859927:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.859934:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.859936:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000100:00100000:7.0:1496184935.859946:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57264 00000100:00000040:7.0:1496184935.859948:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.859950:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.859957:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.860001:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.870120:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184935.881119:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184935.892155:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.892212:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.892217:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.892220:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.892225:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.892228:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.892234:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32277us (32371us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.892239:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57264 00000100:00000040:7.0:1496184935.892241:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.892245:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.892255:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.892257:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000020:00000040:7.0:1496184935.892261:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:1.0:1496184935.902883:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d050 x1568861104093328 msgsize 520 00000100:00100000:1.0:1496184935.902898:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184935.902900:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184935.902916:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093328 00000020:00000040:2.0:1496184935.902921:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184935.902923:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184935.902943:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6029 00000100:00000040:2.0:1496184935.902945:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.902948:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d050 x1568861104093328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184935.902956:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093328:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184935.902965:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184935.902974:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184935.902978:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184935.902983:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:2.0:1496184935.902988:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184935.902991:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184935.902992:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184935.902998:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184935.903002:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.903004:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.903005:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184935.903005:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184935.903008:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.903012:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.903014:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184935.903017:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.903019:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184935.903023:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184935.903026:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184935.903028:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184935.903029:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184935.903035:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093328 00000100:00000040:2.0:1496184935.903041:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.903044:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.903061:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184935.903063:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d050 x1568861104093328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184935.903070:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093328:12345-10.37.248.124@o2ib1:8 Request procesed in 114us (186us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184935.903077:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6029 00000100:00000040:2.0:1496184935.903079:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184935.903084:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.903601:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.903604:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.903606:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.903611:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.903619:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.903626:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.903628:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000100:00100000:7.0:1496184935.903638:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57265 00000100:00000040:7.0:1496184935.903640:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.903642:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.903649:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.903691:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.905348:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184935.905373:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184935.905415:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.905460:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.905465:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.905468:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.905473:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.905476:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.905486:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1837us (1930us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.905491:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57265 00000100:00000040:7.0:1496184935.905493:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.905497:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.905502:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.905505:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000020:00000040:7.0:1496184935.905508:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:2.0:1496184935.933827:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee6850 x1568861152397552 msgsize 224 00000100:00100000:2.0:1496184935.933834:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184935.933836:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184935.933888:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397552 00000020:00000040:1.0:1496184935.933912:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:1.0:1496184935.933914:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:1.0:1496184935.933921:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184935.933934:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 954 00000100:00000040:1.0:1496184935.933936:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:1.0:1496184935.933939:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee6850 x1568861152397552/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:686/0 lens 224/0 e 0 to 0 dl 1496184941 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184935.933946:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397552:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184935.933963:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:1.0:1496184935.933968:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397552 00000100:00000040:1.0:1496184935.933973:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184935.933976:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184935.933993:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184935.933995:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee6850 x1568861152397552/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:686/0 lens 224/336 e 0 to 0 dl 1496184941 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184935.934001:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397552:12345-10.37.248.197@o2ib1:13 Request procesed in 55us (173us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184935.934006:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 954 00000100:00000040:1.0:1496184935.934007:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:1.0:1496184935.934011:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00000100:00100000:7.0:1496184935.934449:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093360 00000020:00000040:7.0:1496184935.934453:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.934454:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.934469:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43075 00000100:00000040:7.0:1496184935.934471:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.934473:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2c50 x1568861104093360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.934481:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093360:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.934487:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.934492:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.934495:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.934498:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.934502:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.934504:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.934505:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.934510:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.934513:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.934514:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.934515:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.934516:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.934519:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.934523:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.934524:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.934527:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.934529:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.934532:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.934534:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.934536:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.934537:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.934541:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093360 00000100:00000040:7.0:1496184935.934545:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.934548:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.934559:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.934562:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2c50 x1568861104093360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.934568:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093360:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.934572:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43075 00000100:00000040:7.0:1496184935.934574:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.934577:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.935185:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.935188:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.935190:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.935195:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.935203:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.935209:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.935212:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000100:00100000:7.0:1496184935.935222:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57266 00000100:00000040:7.0:1496184935.935223:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.935226:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.935232:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.935274:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.936834:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184935.936859:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184935.936912:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.936963:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.936967:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.936970:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.936975:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.936978:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.936984:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1752us (1845us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.936989:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57266 00000100:00000040:7.0:1496184935.936991:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.936994:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.937000:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.937003:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000020:00000040:7.0:1496184935.937006:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.966162:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093392 00000020:00000040:7.0:1496184935.966165:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.966167:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.966181:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43076 00000100:00000040:7.0:1496184935.966183:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.966185:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2850 x1568861104093392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.966193:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093392:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.966198:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.966203:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.966206:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.966208:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.966212:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.966214:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.966215:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.966220:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.966223:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.966224:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.966225:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.966226:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.966228:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.966232:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.966233:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.966235:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.966237:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.966240:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.966243:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.966244:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.966245:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.966249:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093392 00000100:00000040:7.0:1496184935.966253:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.966256:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.966268:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.966270:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2850 x1568861104093392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.966276:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093392:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.966281:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43076 00000100:00000040:7.0:1496184935.966282:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.966285:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.966774:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.966777:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.966778:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.966783:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.966791:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.966798:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.966800:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:7.0:1496184935.966810:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57267 00000100:00000040:7.0:1496184935.966812:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.966814:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.966820:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.966864:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.968439:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d200 00000100:00020000:3.0:1496184935.968464:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d200 00010000:00020000:7.0:1496184935.968508:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184935.968554:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184935.968558:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184935.968561:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184935.968566:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184935.968569:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184935.968575:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1754us (1845us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184935.968580:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57267 00000100:00000040:7.0:1496184935.968581:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184935.968585:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184935.968591:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.968593:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:7.0:1496184935.968596:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.997086:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093424 00000020:00000040:7.0:1496184935.997089:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.997090:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184935.997104:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43077 00000100:00000040:7.0:1496184935.997106:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.997109:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1c50 x1568861104093424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184955 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.997116:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093424:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184935.997122:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184935.997127:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184935.997130:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184935.997133:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184935 last 1496184935 00000001:00000040:7.0:1496184935.997136:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184935.997138:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184935.997140:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184935.997144:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184935.997147:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.997148:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.997149:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184935.997150:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.997152:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.997156:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.997157:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184935.997160:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.997161:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184935.997164:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184935.997166:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184935.997168:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184935.997169:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184935.997173:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093424 00000100:00000040:7.0:1496184935.997177:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.997179:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.997191:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184935.997194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1c50 x1568861104093424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184955 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184935.997200:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093424:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184935.997204:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43077 00000100:00000040:7.0:1496184935.997206:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184935.997209:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184935.997744:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184935.997748:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184935.997749:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184935.997755:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184935.997763:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184935.997770:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184935.997772:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:7.0:1496184935.997782:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57268 00000100:00000040:7.0:1496184935.997783:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184935.997785:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/0 e 0 to 0 dl 1496184978 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184935.997792:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184935.997834:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184935.999402:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184936.010553:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00010000:00020000:7.0:1496184936.021645:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.021690:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.021695:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.021698:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.021703:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.021706:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.021712:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23920us (24013us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.021717:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57268 00000100:00000040:7.0:1496184936.021719:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.021722:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:723/0 lens 608/448 e 0 to 0 dl 1496184978 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.021728:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.021730:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000020:00000040:7.0:1496184936.021734:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.032135:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093456 00000020:00000040:7.0:1496184936.032138:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.032140:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.032154:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43078 00000100:00000040:7.0:1496184936.032156:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.032158:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2050 x1568861104093456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.032166:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093456:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.032171:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.032176:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.032179:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.032182:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.032185:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.032187:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.032189:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.032194:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.032196:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.032198:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.032199:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.032200:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.032202:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.032205:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.032207:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.032209:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.032211:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.032214:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.032216:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.032218:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.032219:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.032222:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093456 00000100:00000040:7.0:1496184936.032226:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.032229:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.032241:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.032243:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2050 x1568861104093456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.032249:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093456:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.032254:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43078 00000100:00000040:7.0:1496184936.032255:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.032258:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.032813:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.032816:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.032817:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.032823:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.032831:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.032837:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.032840:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:7.0:1496184936.032849:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57269 00000100:00000040:7.0:1496184936.032851:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.032853:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.032860:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.032918:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.034607:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184936.045661:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00010000:00020000:7.0:1496184936.056770:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184936.056790:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d850 x1568861104093488 msgsize 520 00000100:00100000:1.0:1496184936.056799:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184936.056800:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184936.056827:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.056832:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.056835:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.056840:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.056843:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.056849:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23989us (24082us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.056854:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57269 00000100:00100000:2.0:1496184936.056854:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093488 00000100:00000040:7.0:1496184936.056855:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.056859:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184936.056859:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184936.056862:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00100000:7.0:1496184936.056869:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.056872:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:7.0:1496184936.056876:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184936.056900:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6030 00000100:00000040:2.0:1496184936.056903:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.056908:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d850 x1568861104093488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184936.056924:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093488:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184936.056936:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184936.056949:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184936.056953:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184936.056957:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:2.0:1496184936.056966:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184936.056969:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184936.056971:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184936.056985:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184936.056990:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.056992:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.056993:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184936.056994:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.056999:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.057005:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.057008:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184936.057011:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.057013:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.057018:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184936.057021:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184936.057024:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.057026:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184936.057034:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093488 00000100:00000040:2.0:1496184936.057041:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.057045:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.057073:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.057076:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d850 x1568861104093488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184936.057083:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093488:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (291us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184936.057088:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6030 00000100:00000040:2.0:1496184936.057090:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.057095:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.058004:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.058007:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.058009:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.058014:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.058022:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.058029:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.058031:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000100:00100000:7.0:1496184936.058041:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57270 00000100:00000040:7.0:1496184936.058043:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.058045:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.058052:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.058093:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.070168:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184936.081163:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00010000:00020000:7.0:1496184936.092196:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.092253:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.092258:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.092261:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.092266:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.092269:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.092279:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34227us (34323us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.092284:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57270 00000100:00000040:7.0:1496184936.092286:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.092289:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.092295:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.092298:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000020:00000040:7.0:1496184936.092301:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.103094:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093520 00000020:00000040:7.0:1496184936.103097:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.103099:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.103113:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43079 00000100:00000040:7.0:1496184936.103115:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.103118:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1850 x1568861104093520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.103125:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093520:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.103131:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.103136:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.103139:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.103141:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.103145:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.103147:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.103149:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.103153:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.103157:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.103158:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.103159:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.103160:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.103162:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.103166:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.103168:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.103170:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.103172:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.103175:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.103177:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.103179:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.103181:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.103184:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093520 00000100:00000040:7.0:1496184936.103188:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.103191:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.103203:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.103205:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1850 x1568861104093520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.103211:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093520:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (171us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.103216:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43079 00000100:00000040:7.0:1496184936.103217:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.103220:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.103719:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.103722:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.103724:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.103730:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.103738:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.103745:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.103747:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000100:00100000:7.0:1496184936.103757:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57271 00000100:00000040:7.0:1496184936.103759:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.103761:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.103768:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.103810:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.105417:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184936.105428:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184936.105470:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.105516:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.105520:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.105523:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.105528:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.105531:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.105537:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1769us (1860us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.105542:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57271 00000100:00000040:7.0:1496184936.105543:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.105547:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.105553:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.105555:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000020:00000040:7.0:1496184936.105559:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.133950:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093552 00000020:00000040:7.0:1496184936.133953:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.133954:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.133969:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43080 00000100:00000040:7.0:1496184936.133970:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.133973:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0450 x1568861104093552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.133980:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093552:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.133986:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.133991:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.133994:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.133997:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.134000:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.134002:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.134004:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.134008:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.134011:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.134012:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.134013:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.134014:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.134016:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.134020:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.134021:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.134024:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.134025:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.134028:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.134031:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.134032:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.134034:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.134037:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093552 00000100:00000040:7.0:1496184936.134041:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.134044:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.134056:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.134059:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0450 x1568861104093552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.134065:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093552:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.134069:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43080 00000100:00000040:7.0:1496184936.134071:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.134074:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.134622:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.134625:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.134627:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.134632:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.134640:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.134647:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.134649:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000100:00100000:7.0:1496184936.134659:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57272 00000100:00000040:7.0:1496184936.134660:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.134663:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.134670:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.134713:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.136313:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184936.136328:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184936.136373:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.136419:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.136423:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.136426:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.136432:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.136434:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.136440:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1770us (1862us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.136445:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57272 00000100:00000040:7.0:1496184936.136447:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.136450:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.136456:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.136458:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000020:00000040:7.0:1496184936.136462:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.164516:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093584 00000020:00000040:7.0:1496184936.164519:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.164521:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.164535:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43081 00000100:00000040:7.0:1496184936.164537:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.164539:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1450 x1568861104093584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.164547:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093584:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.164553:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.164558:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.164561:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.164563:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.164567:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.164569:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.164570:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.164574:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.164577:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.164579:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.164579:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.164580:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.164582:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.164586:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.164588:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.164590:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.164592:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.164594:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.164597:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.164598:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.164599:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.164603:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093584 00000100:00000040:7.0:1496184936.164607:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.164610:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.164622:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.164624:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1450 x1568861104093584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.164630:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093584:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.164635:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43081 00000100:00000040:7.0:1496184936.164636:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.164639:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.165192:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.165195:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.165197:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.165202:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.165210:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.165216:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.165219:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000100:00100000:7.0:1496184936.165228:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57273 00000100:00000040:7.0:1496184936.165230:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.165232:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.165239:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.165280:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.167030:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184936.167037:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184936.167087:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.167133:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.167137:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.167140:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.167145:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.167148:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.167154:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1915us (2004us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.167159:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57273 00000100:00000040:7.0:1496184936.167161:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.167164:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.167170:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.167172:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000020:00000040:7.0:1496184936.167176:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.194721:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093616 00000020:00000040:7.0:1496184936.194724:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.194725:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.194740:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43082 00000100:00000040:7.0:1496184936.194741:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.194744:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0850 x1568861104093616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.194751:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093616:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.194757:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.194762:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.194765:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.194767:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.194771:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.194773:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.194774:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.194779:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.194782:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.194783:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.194784:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.194785:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.194787:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.194791:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.194792:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.194795:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.194796:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.194799:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.194801:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.194803:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.194804:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.194808:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093616 00000100:00000040:7.0:1496184936.194812:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.194814:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.194826:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.194829:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0850 x1568861104093616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.194835:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093616:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (158us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.194839:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43082 00000100:00000040:7.0:1496184936.194841:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.194844:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.195394:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.195397:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.195398:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.195404:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.195412:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.195418:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.195421:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000100:00100000:7.0:1496184936.195430:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57274 00000100:00000040:7.0:1496184936.195432:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.195434:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.195440:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.195482:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.196983:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184936.208132:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184936.219211:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.219269:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.219274:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.219277:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.219282:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.219285:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.219291:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23850us (23942us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.219296:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57274 00000100:00000040:7.0:1496184936.219297:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.219301:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.219311:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.219314:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000020:00000040:7.0:1496184936.219317:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.229648:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093648 00000020:00000040:7.0:1496184936.229651:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.229657:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.229671:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43083 00000100:00000040:7.0:1496184936.229673:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.229675:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1050 x1568861104093648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.229683:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093648:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.229688:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.229693:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.229696:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.229699:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.229702:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.229704:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.229706:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.229710:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.229713:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.229714:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.229715:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.229716:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.229718:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.229722:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.229724:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.229726:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.229728:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.229730:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.229733:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.229734:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.229735:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.229739:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093648 00000100:00000040:7.0:1496184936.229743:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.229746:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.229757:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.229760:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1050 x1568861104093648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.229766:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093648:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.229770:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43083 00000100:00000040:7.0:1496184936.229772:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.229775:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.230329:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.230332:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.230334:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.230340:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.230348:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.230354:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.230357:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000100:00100000:7.0:1496184936.230366:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57275 00000100:00000040:7.0:1496184936.230368:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.230371:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.230377:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.230421:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.231951:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184936.243020:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184936.254122:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184936.254143:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c850 x1568861104093680 msgsize 520 00000100:00100000:1.0:1496184936.254151:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184936.254153:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184936.254168:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.254173:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.254176:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184936.254177:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093680 00010000:00000040:7.0:1496184936.254181:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000040:2.0:1496184936.254182:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000100:00000040:7.0:1496184936.254183:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184936.254185:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:7.0:1496184936.254189:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23812us (23902us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.254194:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57275 00000100:00000040:7.0:1496184936.254196:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.254200:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.254206:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.254208:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000020:00000040:7.0:1496184936.254212:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184936.254213:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6031 00000100:00000040:2.0:1496184936.254216:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.254221:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c850 x1568861104093680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184936.254229:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093680:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184936.254244:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184936.254263:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184936.254268:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184936.254272:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:2.0:1496184936.254280:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184936.254284:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184936.254286:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184936.254294:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184936.254299:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.254301:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.254302:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184936.254303:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.254306:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.254312:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.254315:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184936.254318:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.254320:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.254325:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184936.254328:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184936.254330:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.254332:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184936.254341:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093680 00000100:00000040:2.0:1496184936.254348:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.254352:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.254371:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.254374:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c850 x1568861104093680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184936.254381:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093680:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (238us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184936.254386:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6031 00000100:00000040:2.0:1496184936.254388:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.254401:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.255353:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.255357:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.255358:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.255371:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.255378:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.255385:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.255388:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000100:00100000:7.0:1496184936.255398:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57276 00000100:00000040:7.0:1496184936.255399:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.255402:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.255408:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.255452:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.257233:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184936.268236:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184936.279266:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.279324:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.279328:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.279331:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.279337:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.279339:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.279345:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23937us (24037us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.279350:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57276 00000100:00000040:7.0:1496184936.279352:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.279355:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.279361:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.279364:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000020:00000040:7.0:1496184936.279367:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.291692:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093712 00000020:00000040:7.0:1496184936.291695:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.291696:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.291711:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43084 00000100:00000040:7.0:1496184936.291712:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.291715:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0c50 x1568861104093712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.291722:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093712:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.291728:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.291734:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.291736:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.291739:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.291743:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.291745:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.291746:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.291751:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.291754:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.291755:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.291756:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.291757:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.291759:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.291764:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.291765:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.291768:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.291770:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.291773:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.291775:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.291777:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.291778:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.291782:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093712 00000100:00000040:7.0:1496184936.291786:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.291789:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.291801:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.291804:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0c50 x1568861104093712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.291810:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093712:12345-10.37.248.124@o2ib1:8 Request procesed in 88us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.291814:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43084 00000100:00000040:7.0:1496184936.291816:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.291819:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.292366:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.292369:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.292371:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.292376:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.292384:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.292390:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.292393:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000100:00100000:7.0:1496184936.292403:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57277 00000100:00000040:7.0:1496184936.292404:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.292407:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.292413:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.292456:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.294162:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90cc00 00000100:00020000:3.0:1496184936.305194:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90cc00 00010000:00020000:7.0:1496184936.316247:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.316292:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.316297:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.316300:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.316305:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.316308:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.316313:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23900us (23993us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.316318:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57277 00000100:00000040:7.0:1496184936.316320:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.316324:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.316329:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.316332:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000020:00000040:7.0:1496184936.316335:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.326647:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093744 00000020:00000040:7.0:1496184936.326650:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.326652:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.326667:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43085 00000100:00000040:7.0:1496184936.326669:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.326671:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd8c50 x1568861104093744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.326679:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093744:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.326685:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.326690:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.326693:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.326695:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.326699:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.326701:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.326702:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.326707:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.326710:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.326711:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.326712:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.326713:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.326715:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.326719:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.326720:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.326723:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.326725:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.326727:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.326729:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.326731:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.326732:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.326736:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093744 00000100:00000040:7.0:1496184936.326740:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.326743:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.326756:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.326758:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd8c50 x1568861104093744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.326765:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093744:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.326769:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43085 00000100:00000040:7.0:1496184936.326770:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.326774:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.327418:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.327422:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.327425:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.327432:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.327444:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.327454:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.327458:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000100:00100000:7.0:1496184936.327480:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57278 00000100:00000040:7.0:1496184936.327482:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.327484:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.327491:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.327534:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.329281:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184936.340346:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184936.351347:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.351408:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.351413:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.351416:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.351421:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.351423:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.351429:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23939us (24076us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.351434:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57278 00000100:00000040:7.0:1496184936.351436:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.351440:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.351445:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.351448:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000020:00000040:7.0:1496184936.351451:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:1.0:1496184936.362102:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e450 x1568861104093776 msgsize 520 00000100:00100000:1.0:1496184936.362110:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184936.362112:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184936.362141:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093776 00000020:00000040:2.0:1496184936.362146:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184936.362149:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184936.362174:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6032 00000100:00000040:2.0:1496184936.362176:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.362181:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e450 x1568861104093776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184936.362190:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093776:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184936.362202:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184936.362215:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184936.362219:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184936.362223:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:2.0:1496184936.362231:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184936.362234:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184936.362237:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184936.362245:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184936.362249:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.362251:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.362252:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184936.362253:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184936.362257:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.362265:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.362268:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184936.362271:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.362273:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184936.362279:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184936.362281:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184936.362284:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184936.362285:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184936.362294:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093776 00000100:00000040:2.0:1496184936.362301:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.362305:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.362325:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184936.362328:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e450 x1568861104093776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184936.362334:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093776:12345-10.37.248.124@o2ib1:8 Request procesed in 145us (232us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184936.362339:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6032 00000100:00000040:2.0:1496184936.362341:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184936.362347:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.362880:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.362899:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.362901:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.362906:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.362912:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.362925:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.362927:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000100:00100000:7.0:1496184936.362934:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57279 00000100:00000040:7.0:1496184936.362935:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.362938:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.362943:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.362986:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.364652:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184936.375691:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184936.386717:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.386777:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.386780:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.386783:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.386787:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.386789:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.386795:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23851us (23941us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.386799:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57279 00000100:00000040:7.0:1496184936.386800:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.386803:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.386808:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.386809:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000020:00000040:7.0:1496184936.386812:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.397795:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093808 00000020:00000040:7.0:1496184936.397799:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.397800:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.397815:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43086 00000100:00000040:7.0:1496184936.397816:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.397819:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c0050 x1568861104093808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.397826:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093808:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.397832:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.397838:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.397841:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.397843:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.397847:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.397849:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.397851:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.397855:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.397858:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.397860:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.397860:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.397861:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.397864:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.397868:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.397869:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.397872:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.397874:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.397876:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.397879:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.397881:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.397882:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.397886:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093808 00000100:00000040:7.0:1496184936.397901:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.397904:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.397916:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.397918:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c0050 x1568861104093808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.397925:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093808:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (178us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.397929:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43086 00000100:00000040:7.0:1496184936.397931:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.397934:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.398549:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.398552:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.398554:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.398559:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.398567:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.398573:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.398576:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000100:00100000:7.0:1496184936.398586:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57280 00000100:00000040:7.0:1496184936.398587:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.398590:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.398596:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.398639:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.400191:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184936.411234:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184936.422302:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.422347:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.422352:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.422355:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.422360:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.422362:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.422372:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23776us (23877us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.422377:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57280 00000100:00000040:7.0:1496184936.422379:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.422383:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.422388:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.422391:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000020:00000040:7.0:1496184936.422394:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.432782:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093840 00000020:00000040:7.0:1496184936.432785:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.432786:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.432801:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43087 00000100:00000040:7.0:1496184936.432802:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.432805:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9450 x1568861104093840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.432812:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093840:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.432818:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.432823:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.432826:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.432828:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.432832:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.432834:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.432835:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.432840:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.432843:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.432844:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.432845:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.432846:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.432848:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.432852:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.432854:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.432856:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.432858:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.432860:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.432863:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.432864:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.432866:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.432869:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093840 00000100:00000040:7.0:1496184936.432873:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.432876:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.432888:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.432903:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9450 x1568861104093840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.432909:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093840:12345-10.37.248.124@o2ib1:8 Request procesed in 97us (172us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.432914:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43087 00000100:00000040:7.0:1496184936.432915:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.432919:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.433471:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.433474:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.433476:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.433481:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.433489:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.433495:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.433498:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000100:00100000:7.0:1496184936.433508:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57281 00000100:00000040:7.0:1496184936.433509:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.433512:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.433518:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.433561:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.434945:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184936.445992:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00000100:00000040:2.0:1496184936.457050:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d450 x1568861104093872 msgsize 520 00000100:00100000:2.0:1496184936.457058:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184936.457060:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184936.457081:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093872 00000020:00000040:3.0:1496184936.457085:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00010000:00020000:7.0:1496184936.457097:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:3.0:1496184936.457097:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184936.457122:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6033 00000100:00000040:3.0:1496184936.457124:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184936.457129:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d450 x1568861104093872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184936.457140:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104093872:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.457143:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:7.0:1496184936.457148:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.457151:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184936.457152:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00010000:00000040:7.0:1496184936.457156:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.457158:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.457165:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23646us (23738us total) trans 0 rc -110/-999 00000020:00000040:3.0:1496184936.457165:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:7.0:1496184936.457170:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57281 00010000:02000400:3.0:1496184936.457170:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00000100:00000040:7.0:1496184936.457172:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00010000:00080000:3.0:1496184936.457174:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00002000:00100000:7.0:1496184936.457176:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.457181:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00000001:00000040:3.0:1496184936.457182:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00002000:00010000:7.0:1496184936.457185:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000001:00000040:3.0:1496184936.457185:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184936.457187:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00000020:00000040:7.0:1496184936.457189:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00002000:00100000:3.0:1496184936.457196:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184936.457200:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.457202:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.457203:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184936.457204:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.457208:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.457213:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.457216:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184936.457219:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.457221:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.457226:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184936.457236:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184936.457239:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.457240:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184936.457249:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093872 00000100:00000040:3.0:1496184936.457256:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.457260:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.457280:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.457283:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d450 x1568861104093872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184936.457289:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093872:12345-10.37.248.124@o2ib1:8 Request procesed in 149us (239us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184936.457295:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6033 00000100:00000040:3.0:1496184936.457297:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.457303:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.458292:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.458296:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.458298:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.458304:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.458311:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.458318:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.458321:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000100:00100000:7.0:1496184936.458330:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57282 00000100:00000040:7.0:1496184936.458332:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.458335:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.458341:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.458382:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.470241:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184936.481239:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184936.492282:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.492340:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.492345:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.492348:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.492353:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.492355:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.492361:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34020us (34116us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.492369:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57282 00000100:00000040:7.0:1496184936.492371:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.492374:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.492380:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.492382:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000020:00000040:7.0:1496184936.492386:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.503080:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093904 00000020:00000040:7.0:1496184936.503083:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.503085:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.503100:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43088 00000100:00000040:7.0:1496184936.503102:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.503104:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9050 x1568861104093904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.503112:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093904:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.503118:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.503123:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.503126:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.503128:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.503132:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.503134:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.503136:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.503140:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.503143:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.503145:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.503146:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.503147:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.503149:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.503154:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.503155:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.503158:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.503160:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.503162:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.503165:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.503167:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.503168:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.503171:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093904 00000100:00000040:7.0:1496184936.503176:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.503178:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.503192:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.503194:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9050 x1568861104093904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.503200:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093904:12345-10.37.248.124@o2ib1:8 Request procesed in 89us (167us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.503205:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43088 00000100:00000040:7.0:1496184936.503206:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.503209:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.503735:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.503738:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.503740:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.503746:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.503754:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.503760:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.503763:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000100:00100000:7.0:1496184936.503773:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57283 00000100:00000040:7.0:1496184936.503775:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.503777:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.503783:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.503826:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.505525:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184936.505536:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184936.505582:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.505627:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.505631:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.505635:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.505640:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.505642:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.505648:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1864us (1959us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.505653:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57283 00000100:00000040:7.0:1496184936.505655:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.505658:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.505664:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.505666:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000020:00000040:7.0:1496184936.505670:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.534414:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093936 00000020:00000040:7.0:1496184936.534417:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.534419:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.534433:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43089 00000100:00000040:7.0:1496184936.534435:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.534438:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9c50 x1568861104093936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.534445:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093936:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.534451:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.534456:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.534459:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.534461:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.534465:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.534467:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.534468:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.534472:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.534475:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.534477:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.534478:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.534478:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.534480:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.534484:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.534486:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.534488:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.534490:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.534492:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.534495:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.534497:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.534498:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.534502:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093936 00000100:00000040:7.0:1496184936.534506:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.534508:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.534520:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.534523:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9c50 x1568861104093936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.534529:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093936:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.534533:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43089 00000100:00000040:7.0:1496184936.534535:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.534538:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.535099:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.535102:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.535104:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.535109:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.535117:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.535124:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.535126:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000100:00100000:7.0:1496184936.535136:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57284 00000100:00000040:7.0:1496184936.535138:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.535140:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.535147:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.535190:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.536650:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184936.536662:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184936.536715:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.536760:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.536765:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.536768:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.536773:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.536775:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.536781:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1634us (1726us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.536786:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57284 00000100:00000040:7.0:1496184936.536788:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.536792:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.536797:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.536800:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000020:00000040:7.0:1496184936.536803:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.565331:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104093968 00000020:00000040:7.0:1496184936.565334:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.565336:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.565350:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43090 00000100:00000040:7.0:1496184936.565352:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.565354:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9850 x1568861104093968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.565362:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093968:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.565368:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.565373:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.565376:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.565378:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.565382:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.565384:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.565385:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.565390:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.565393:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.565394:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.565395:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.565396:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.565397:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.565402:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.565403:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.565405:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.565407:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.565410:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.565412:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.565414:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.565415:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.565418:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104093968 00000100:00000040:7.0:1496184936.565423:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.565425:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.565437:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.565439:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9850 x1568861104093968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.565445:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104093968:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (157us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.565450:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43090 00000100:00000040:7.0:1496184936.565452:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.565455:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.566004:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.566007:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.566008:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.566014:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.566021:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.566028:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.566030:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000100:00100000:7.0:1496184936.566040:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57285 00000100:00000040:7.0:1496184936.566042:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.566044:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.566051:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.566094:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.567609:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184936.567624:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184936.567677:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.567723:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.567727:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.567731:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.567736:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.567738:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.567748:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1697us (1789us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.567753:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57285 00000100:00000040:7.0:1496184936.567755:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.567758:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.567764:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.567767:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000020:00000040:7.0:1496184936.567770:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.595136:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094000 00000020:00000040:7.0:1496184936.595139:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.595141:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.595155:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43091 00000100:00000040:7.0:1496184936.595157:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.595159:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda850 x1568861104094000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.595167:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094000:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.595173:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.595178:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.595181:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.595183:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.595187:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.595188:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.595190:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.595194:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.595197:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.595199:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.595200:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.595201:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.595203:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.595207:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.595208:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.595211:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.595212:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.595215:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.595217:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.595219:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.595220:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.595224:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094000 00000100:00000040:7.0:1496184936.595228:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.595231:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.595242:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.595245:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda850 x1568861104094000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.595251:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094000:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.595255:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43091 00000100:00000040:7.0:1496184936.595257:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.595260:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.595835:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.595838:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.595840:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.595846:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.595854:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.595861:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.595863:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000100:00100000:7.0:1496184936.595873:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57286 00000100:00000040:7.0:1496184936.595874:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.595877:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.595883:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.595947:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.597515:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ee00 00000100:00020000:3.0:1496184936.608645:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ee00 00010000:00020000:7.0:1496184936.619722:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.619768:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.619772:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.619776:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.619780:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.619783:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.619789:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23905us (23998us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.619794:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57286 00000100:00000040:7.0:1496184936.619796:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.619799:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.619805:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.619808:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000020:00000040:7.0:1496184936.619811:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.630280:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094032 00000020:00000040:7.0:1496184936.630283:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.630285:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.630299:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43092 00000100:00000040:7.0:1496184936.630301:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.630304:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda450 x1568861104094032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.630311:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094032:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.630317:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.630322:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.630325:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.630327:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.630331:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.630333:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.630334:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.630339:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.630342:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.630343:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.630344:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.630345:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.630347:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.630351:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.630352:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.630355:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.630356:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.630359:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.630362:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.630363:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.630365:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.630368:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094032 00000100:00000040:7.0:1496184936.630372:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.630375:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.630387:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.630389:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda450 x1568861104094032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.630395:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094032:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.630400:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43092 00000100:00000040:7.0:1496184936.630401:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.630405:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.630972:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.630975:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.630977:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.630982:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.630990:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.630996:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.630999:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000100:00100000:7.0:1496184936.631009:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57287 00000100:00000040:7.0:1496184936.631010:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.631013:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.631019:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.631062:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.632714:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184936.643778:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184936.654916:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184936.654924:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8dc50 x1568861104094064 msgsize 520 00000100:00100000:1.0:1496184936.654933:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184936.654935:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184936.654962:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.654966:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.654970:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.654975:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.654977:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.654983:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23964us (24055us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184936.654985:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094064 00000100:00100000:7.0:1496184936.654988:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57287 00000100:00000040:7.0:1496184936.654990:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.654990:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00100000:7.0:1496184936.654993:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184936.654993:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00100000:7.0:1496184936.654999:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.655002:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000020:00000040:7.0:1496184936.655005:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184936.655022:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6034 00000100:00000040:3.0:1496184936.655024:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.655030:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8dc50 x1568861104094064/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184936.655038:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094064:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184936.655057:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184936.655072:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184936.655077:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184936.655081:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:3.0:1496184936.655089:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184936.655093:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184936.655095:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184936.655103:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184936.655108:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.655110:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.655111:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184936.655112:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.655115:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.655121:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.655123:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184936.655126:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.655129:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.655134:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184936.655137:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184936.655139:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.655141:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184936.655149:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094064 00000100:00000040:3.0:1496184936.655157:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.655160:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.655179:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.655182:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8dc50 x1568861104094064/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184936.655188:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094064:12345-10.37.248.124@o2ib1:8 Request procesed in 150us (263us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184936.655194:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6034 00000100:00000040:3.0:1496184936.655204:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.655209:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.656102:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.656105:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.656107:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.656113:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.656120:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.656127:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.656130:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000100:00100000:7.0:1496184936.656139:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57288 00000100:00000040:7.0:1496184936.656141:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.656143:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.656150:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.656193:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.657860:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184936.668862:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184936.679924:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.679969:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.679973:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.679977:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.679982:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.679984:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.679990:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23840us (23933us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.679995:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57288 00000100:00000040:7.0:1496184936.679997:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.680001:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.680006:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.680009:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000020:00000040:7.0:1496184936.680012:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.691665:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094096 00000020:00000040:7.0:1496184936.691668:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.691670:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.691684:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43093 00000100:00000040:7.0:1496184936.691686:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.691689:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdac50 x1568861104094096/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.691696:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094096:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.691702:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.691707:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.691710:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.691713:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.691717:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.691719:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.691720:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.691725:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.691728:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.691729:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.691730:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.691731:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.691733:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.691738:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.691739:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.691742:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.691744:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.691746:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.691749:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.691751:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.691752:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.691755:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094096 00000100:00000040:7.0:1496184936.691760:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.691762:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.691774:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.691776:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdac50 x1568861104094096/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.691782:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094096:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.691787:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43093 00000100:00000040:7.0:1496184936.691788:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.691792:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.692326:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.692329:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.692330:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.692336:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.692343:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.692350:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.692353:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000100:00100000:7.0:1496184936.692362:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57289 00000100:00000040:7.0:1496184936.692364:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.692366:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.692373:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.692415:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.693968:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184936.704999:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184936.716063:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.716108:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.716113:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.716116:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.716121:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.716124:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.716130:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23756us (23855us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.716135:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57289 00000100:00000040:7.0:1496184936.716136:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.716140:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.716145:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.716148:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000020:00000040:7.0:1496184936.716151:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.726753:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094128 00000020:00000040:7.0:1496184936.726756:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.726758:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.726772:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43094 00000100:00000040:7.0:1496184936.726774:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.726777:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb050 x1568861104094128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.726784:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094128:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.726790:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.726795:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.726798:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.726800:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.726804:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.726806:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.726807:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.726812:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.726815:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.726816:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.726817:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.726818:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.726820:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.726824:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.726825:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.726827:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.726829:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.726832:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.726834:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.726836:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.726837:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.726841:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094128 00000100:00000040:7.0:1496184936.726845:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.726847:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.726859:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.726862:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb050 x1568861104094128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.726868:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094128:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (158us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.726872:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43094 00000100:00000040:7.0:1496184936.726874:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.726877:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.727538:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.727542:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.727543:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.727548:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.727556:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.727563:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.727565:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000100:00100000:7.0:1496184936.727575:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57290 00000100:00000040:7.0:1496184936.727576:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.727579:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.727585:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.727628:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.729439:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184936.740499:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00010000:00020000:7.0:1496184936.751571:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.751628:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.751633:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.751636:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.751641:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.751644:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.751654:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24068us (24161us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.751659:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57290 00000100:00000040:7.0:1496184936.751660:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.751664:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.751670:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.751672:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000020:00000040:7.0:1496184936.751675:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.762277:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094160 00000020:00000040:7.0:1496184936.762280:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.762282:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.762296:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43095 00000100:00000040:7.0:1496184936.762298:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.762301:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdbc50 x1568861104094160/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.762308:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094160:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.762314:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.762319:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.762322:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.762324:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.762328:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.762330:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.762331:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.762336:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.762339:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.762340:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.762341:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.762342:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.762344:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.762348:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.762349:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.762351:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.762353:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.762356:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.762358:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.762360:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.762361:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.762364:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094160 00000100:00000040:7.0:1496184936.762369:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.762371:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.762383:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.762385:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdbc50 x1568861104094160/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.762391:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094160:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (158us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.762396:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43095 00000100:00000040:7.0:1496184936.762397:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.762400:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.762992:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.762995:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.762996:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.763002:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.763010:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.763017:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.763019:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000100:00100000:7.0:1496184936.763029:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57291 00000100:00000040:7.0:1496184936.763031:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.763033:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.763039:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.763082:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.764852:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184936.775887:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184936.786939:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.786998:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.787002:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.787005:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.787010:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.787013:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.787019:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23979us (24071us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.787024:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57291 00000100:00000040:7.0:1496184936.787025:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.787029:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.787035:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.787037:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000020:00000040:7.0:1496184936.787041:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.797783:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094192 00000020:00000040:7.0:1496184936.797788:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.797790:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.797812:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43096 00000100:00000040:7.0:1496184936.797814:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.797818:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb850 x1568861104094192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.797839:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094192:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.797845:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.797850:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.797853:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.797855:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.797859:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.797861:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.797862:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.797867:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.797870:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.797871:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.797872:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.797873:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.797875:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.797879:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.797880:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.797882:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.797884:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.797887:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.797903:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.797905:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.797906:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.797910:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094192 00000100:00000040:7.0:1496184936.797914:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.797917:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.797929:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.797932:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb850 x1568861104094192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.797938:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094192:12345-10.37.248.124@o2ib1:8 Request procesed in 99us (217us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.797942:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43096 00000100:00000040:7.0:1496184936.797944:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.797947:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.798613:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.798616:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.798618:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.798623:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.798631:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.798637:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.798640:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000100:00100000:7.0:1496184936.798650:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57292 00000100:00000040:7.0:1496184936.798651:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.798654:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.798660:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.798703:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.800351:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184936.811389:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184936.822434:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.822492:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.822496:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.822500:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.822505:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.822507:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.822513:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23853us (23948us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.822518:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57292 00000100:00000040:7.0:1496184936.822520:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.822523:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.822529:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.822532:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000020:00000040:7.0:1496184936.822535:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.833120:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094224 00000020:00000040:7.0:1496184936.833123:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.833124:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.833138:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43097 00000100:00000040:7.0:1496184936.833140:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.833143:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb450 x1568861104094224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.833150:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094224:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.833156:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.833161:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.833164:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.833166:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.833170:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.833172:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.833173:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.833178:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.833181:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.833182:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.833183:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.833184:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.833186:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.833190:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.833191:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.833193:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.833195:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.833198:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.833200:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.833202:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.833203:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.833207:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094224 00000100:00000040:7.0:1496184936.833211:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.833213:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.833225:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.833228:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb450 x1568861104094224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.833234:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094224:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.833238:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43097 00000100:00000040:7.0:1496184936.833240:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.833243:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.833806:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.833810:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.833811:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.833817:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.833825:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.833831:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.833834:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000100:00100000:7.0:1496184936.833843:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57293 00000100:00000040:7.0:1496184936.833845:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.833847:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.833854:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.833909:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.835531:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184936.846590:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184936.857673:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184936.857694:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8bc50 x1568861104094256 msgsize 520 00000100:00100000:1.0:1496184936.857702:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184936.857704:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184936.857724:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094256 00000020:00000040:3.0:1496184936.857728:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.857731:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.857732:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:02000400:7.0:1496184936.857736:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.857739:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.857744:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.857747:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.857753:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23899us (23999us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184936.857757:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6035 00000100:00100000:7.0:1496184936.857758:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57293 00000100:00000040:7.0:1496184936.857760:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.857760:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00002000:00100000:7.0:1496184936.857764:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184936.857765:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8bc50 x1568861104094256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.857770:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.857772:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000100:00100000:3.0:1496184936.857774:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104094256:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.857776:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.857787:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184936.857799:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184936.857803:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184936.857807:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:3.0:1496184936.857815:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184936.857821:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184936.857824:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184936.857840:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184936.857844:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.857846:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.857847:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184936.857849:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.857852:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.857857:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.857860:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184936.857863:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.857865:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.857870:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184936.857873:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184936.857876:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.857877:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184936.857886:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094256 00000100:00000040:3.0:1496184936.857901:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.857905:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.857925:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.857927:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8bc50 x1568861104094256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184936.857934:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094256:12345-10.37.248.124@o2ib1:8 Request procesed in 160us (239us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184936.857939:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6035 00000100:00000040:3.0:1496184936.857941:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.857946:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.858871:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.858874:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.858876:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.858882:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.858903:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.858909:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.858912:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000100:00100000:7.0:1496184936.858922:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57294 00000100:00000040:7.0:1496184936.858923:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.858926:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.858932:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.858976:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.870252:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184936.881258:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184936.892300:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.892363:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.892367:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.892370:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.892375:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.892378:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.892384:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33451us (33555us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.892389:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57294 00000100:00000040:7.0:1496184936.892391:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.892394:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.892400:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.892403:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000020:00000040:7.0:1496184936.892406:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.902952:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094288 00000020:00000040:7.0:1496184936.902955:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.902956:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.902971:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43098 00000100:00000040:7.0:1496184936.902973:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.902975:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc050 x1568861104094288/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.902982:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094288:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.902988:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.902994:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.902997:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.903000:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.903004:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.903006:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.903007:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.903012:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.903015:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.903016:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.903017:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.903018:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.903020:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.903024:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.903026:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.903028:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.903030:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.903033:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.903035:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.903037:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.903038:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.903042:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094288 00000100:00000040:7.0:1496184936.903046:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.903049:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.903065:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.903068:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc050 x1568861104094288/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.903074:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094288:12345-10.37.248.124@o2ib1:8 Request procesed in 91us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.903078:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43098 00000100:00000040:7.0:1496184936.903080:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.903083:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.903596:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.903599:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.903601:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.903606:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.903614:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.903620:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.903623:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00000100:00100000:7.0:1496184936.903633:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57295 00000100:00000040:7.0:1496184936.903634:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.903637:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.903643:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.903685:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.905430:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184936.905455:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184936.905499:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.905544:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.905549:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.905552:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.905557:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.905559:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.905569:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1926us (2018us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.905574:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57295 00000100:00000040:7.0:1496184936.905576:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.905580:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.905585:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.905588:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00000020:00000040:7.0:1496184936.905591:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:0.0:1496184936.933603:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c450 x1568861104094320 msgsize 520 00000100:00100000:0.0:1496184936.933611:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184936.933613:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184936.933659:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094320 00000020:00000040:3.0:1496184936.933663:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184936.933666:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184936.933690:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6036 00000100:00000040:3.0:1496184936.933692:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.933696:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c450 x1568861104094320/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184936.933704:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094320:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184936.933715:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184936.933725:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184936.933729:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184936.933732:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:3.0:1496184936.933740:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184936.933743:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184936.933744:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184936.933752:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184936.933756:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.933757:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.933758:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184936.933759:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184936.933762:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.933767:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.933770:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184936.933775:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.933777:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184936.933782:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184936.933785:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184936.933787:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184936.933788:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184936.933796:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094320 00000100:00000040:3.0:1496184936.933803:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.933807:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.933824:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184936.933826:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c450 x1568861104094320/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184936.933833:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094320:12345-10.37.248.124@o2ib1:8 Request procesed in 129us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184936.933837:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6036 00000100:00000040:3.0:1496184936.933839:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184936.933844:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.934405:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.934408:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.934410:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.934415:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.934423:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.934430:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.934432:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00000100:00100000:7.0:1496184936.934442:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57296 00000100:00000040:7.0:1496184936.934444:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.934446:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.934453:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.934497:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.936123:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ca00 00000100:00020000:3.0:1496184936.936136:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ca00 00010000:00020000:7.0:1496184936.936192:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.936237:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.936242:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.936245:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.936250:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.936253:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.936259:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1806us (1903us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.936264:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57296 00000100:00000040:7.0:1496184936.936266:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.936269:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.936275:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.936278:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00000020:00000040:7.0:1496184936.936281:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.964446:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094352 00000020:00000040:7.0:1496184936.964449:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.964451:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.964465:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43099 00000100:00000040:7.0:1496184936.964467:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.964470:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc450 x1568861104094352/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.964477:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094352:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.964483:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.964488:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.964491:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.964494:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.964498:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.964500:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.964501:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.964506:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.964509:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.964510:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.964511:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.964512:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.964514:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.964518:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.964520:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.964522:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.964524:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.964527:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.964529:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.964531:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.964532:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.964536:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094352 00000100:00000040:7.0:1496184936.964540:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.964543:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.964555:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.964557:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc450 x1568861104094352/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.964563:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094352:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.964568:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43099 00000100:00000040:7.0:1496184936.964569:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.964573:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.965107:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.965110:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.965112:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.965117:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.965125:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.965131:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.965134:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00000100:00100000:7.0:1496184936.965143:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57297 00000100:00000040:7.0:1496184936.965145:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.965148:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.965154:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.965198:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.966800:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184936.966826:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184936.966869:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184936.966929:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184936.966934:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184936.966937:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184936.966942:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184936.966945:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184936.966950:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1796us (1889us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184936.966955:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57297 00000100:00000040:7.0:1496184936.966957:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184936.966961:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184936.966967:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.966970:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00000020:00000040:7.0:1496184936.966973:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.994976:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094384 00000020:00000040:7.0:1496184936.994979:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.994981:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184936.994995:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43100 00000100:00000040:7.0:1496184936.994997:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.994999:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc850 x1568861104094384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184956 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.995007:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094384:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184936.995013:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184936.995018:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184936.995021:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184936.995023:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184936 last 1496184935 00000001:00000040:7.0:1496184936.995027:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184936.995029:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184936.995030:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184936.995034:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184936.995037:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.995039:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.995039:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184936.995040:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.995043:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.995047:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.995048:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184936.995050:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.995052:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184936.995055:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184936.995057:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184936.995059:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184936.995060:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184936.995063:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094384 00000100:00000040:7.0:1496184936.995068:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.995070:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.995082:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184936.995084:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc850 x1568861104094384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184956 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184936.995091:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094384:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184936.995095:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43100 00000100:00000040:7.0:1496184936.995097:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184936.995100:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184936.995681:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184936.995684:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184936.995686:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184936.995691:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184936.995699:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184936.995706:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184936.995708:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00000100:00100000:7.0:1496184936.995718:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57298 00000100:00000040:7.0:1496184936.995720:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184936.995722:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/0 e 0 to 0 dl 1496184979 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184936.995728:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184936.995771:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184936.997281:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184937.008416:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184937.019481:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184937.019540:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184937.019545:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184937.019548:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184937.019553:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184937.019555:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184937.019561:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23833us (23933us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184937.019566:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57298 00000100:00000040:7.0:1496184937.019568:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184937.019571:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:724/0 lens 608/448 e 0 to 0 dl 1496184979 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184937.019577:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184937.019580:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00000020:00000040:7.0:1496184937.019583:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.029916:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094416 00000020:00000040:4.0:1496184937.029919:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.029921:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.029936:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43101 00000100:00000040:4.0:1496184937.029938:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.029940:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdcc50 x1568861104094416/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.029948:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094416:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.029953:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.029958:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.029961:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.029964:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.029968:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.029969:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.029971:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.029975:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.029978:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.029981:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.029982:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.029983:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.029985:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.029989:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.029991:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.029993:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.029995:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.029997:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.030000:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.030002:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.030003:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.030006:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094416 00000100:00000040:4.0:1496184937.030010:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.030013:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.030025:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.030027:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdcc50 x1568861104094416/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.030033:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094416:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (167us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.030038:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43101 00000100:00000040:4.0:1496184937.030039:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.030042:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.030601:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.030604:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.030606:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.030612:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.030620:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.030633:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.030635:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00000100:00100000:4.0:1496184937.030645:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57299 00000100:00000040:4.0:1496184937.030647:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.030649:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.030656:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.030701:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.032409:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb400 00000100:00020000:3.0:1496184937.043468:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb400 00010000:00020000:4.0:1496184937.054574:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184937.054596:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c050 x1568861104094448 msgsize 520 00000100:00100000:1.0:1496184937.054605:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184937.054606:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.054627:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094448 00000020:00000040:3.0:1496184937.054630:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.054631:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.054634:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:02000400:4.0:1496184937.054636:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.054639:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.054644:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.054647:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.054653:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23997us (24105us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.054658:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57299 00000100:00100000:3.0:1496184937.054659:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6037 00000100:00000040:4.0:1496184937.054660:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.054664:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184937.054664:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.054669:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c050 x1568861104094448/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.054670:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.054675:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00000100:00100000:3.0:1496184937.054678:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104094448:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.054679:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.054690:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.054703:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184937.054707:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.054711:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.054720:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.054723:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.054725:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.054740:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.054745:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.054747:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.054748:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184937.054749:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.054752:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.054757:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.054760:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.054763:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.054765:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.054770:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.054773:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.054776:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.054777:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.054785:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094448 00000100:00000040:3.0:1496184937.054793:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.054797:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.054816:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.054819:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c050 x1568861104094448/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.054826:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094448:12345-10.37.248.124@o2ib1:8 Request procesed in 148us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.054831:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6037 00000100:00000040:3.0:1496184937.054833:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.054838:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.055726:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026076450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.055733:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.055735:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.055774:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.055777:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.055779:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.055784:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.055792:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.055799:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.055801:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00000100:00100000:4.0:1496184937.055811:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57300 00000100:00000040:4.0:1496184937.055813:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.055815:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.055822:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.055865:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.057416:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bb400 00000100:00020000:3.0:1496184937.068419:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bb400 00010000:00020000:4.0:1496184937.079459:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.079517:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.079522:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.079525:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.079530:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.079533:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.079539:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23717us (23812us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.079548:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57300 00000100:00000040:4.0:1496184937.079550:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.079554:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.079559:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.079562:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00000020:00000040:4.0:1496184937.079565:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.093004:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094480 00000020:00000040:4.0:1496184937.093007:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.093009:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.093024:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43102 00000100:00000040:4.0:1496184937.093025:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.093028:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd050 x1568861104094480/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.093035:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094480:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.093041:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.093047:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.093049:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.093052:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.093056:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.093058:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.093059:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.093067:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.093070:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.093071:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.093074:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.093075:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.093077:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.093082:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.093083:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.093086:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.093088:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.093090:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.093093:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.093095:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.093096:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.093099:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094480 00000100:00000040:4.0:1496184937.093103:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.093106:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.093118:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.093121:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd050 x1568861104094480/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.093126:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094480:12345-10.37.248.124@o2ib1:8 Request procesed in 91us (172us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.093131:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43102 00000100:00000040:4.0:1496184937.093132:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.093136:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.093676:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.093679:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.093680:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.093686:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.093694:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.093700:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.093705:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00000100:00100000:4.0:1496184937.093715:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57301 00000100:00000040:4.0:1496184937.093716:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.093719:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.093725:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.093766:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184937.095214:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105baa00 00000100:00020000:0.0:1496184937.106253:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105baa00 00010000:00020000:4.0:1496184937.117329:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.117387:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.117392:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.117395:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.117400:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.117403:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.117409:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23683us (23782us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.117413:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57301 00000100:00000040:4.0:1496184937.117415:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.117419:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.117424:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.117427:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00000020:00000040:4.0:1496184937.117430:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.119707:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094512 00000020:00000040:4.0:1496184937.119712:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.119714:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.119729:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43103 00000100:00000040:4.0:1496184937.119730:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.119733:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd450 x1568861104094512/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.119740:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094512:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.119746:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.119751:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.119754:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.119756:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.119760:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.119762:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.119763:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.119768:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.119771:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.119772:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.119773:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.119774:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.119776:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.119780:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.119781:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.119783:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.119785:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.119788:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.119790:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.119793:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.119794:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.119798:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094512 00000100:00000040:4.0:1496184937.119802:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.119804:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.119816:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.119818:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd450 x1568861104094512/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.119824:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094512:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (162us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.119829:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43103 00000100:00000040:4.0:1496184937.119830:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.119833:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.120339:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026074c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.120354:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.120355:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.120391:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.120394:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.120395:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.120401:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.120409:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.120416:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.120418:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00000100:00100000:4.0:1496184937.120428:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57302 00000100:00000040:4.0:1496184937.120429:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.120432:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.120438:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.120479:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184937.122135:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b8000 00000100:00020000:0.0:1496184937.133142:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b8000 00010000:00020000:4.0:1496184937.144202:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.144263:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.144268:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.144271:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.144276:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.144279:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.144285:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23846us (23944us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.144290:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57302 00000100:00000040:4.0:1496184937.144291:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.144295:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.144300:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.144303:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00000020:00000040:4.0:1496184937.144306:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.146269:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd850 x1568861104094544 msgsize 520 00000100:00100000:7.0:1496184937.146276:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.146277:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.146316:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094544 00000020:00000040:4.0:1496184937.146319:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.146321:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.146336:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43104 00000100:00000040:4.0:1496184937.146337:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.146340:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd850 x1568861104094544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.146347:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094544:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.146353:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.146360:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.146363:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.146366:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.146370:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.146372:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.146373:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.146377:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.146381:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.146382:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.146383:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.146384:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.146386:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.146390:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.146391:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.146393:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.146395:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.146398:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.146400:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.146402:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.146403:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.146406:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094544 00000100:00000040:4.0:1496184937.146411:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.146413:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.146425:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.146427:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd850 x1568861104094544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.146433:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094544:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (164us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.146439:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43104 00000100:00000040:4.0:1496184937.146440:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.146443:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.146937:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.146944:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.146945:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.146991:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.146994:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.146995:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.147001:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.147009:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.147015:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.147018:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00000100:00100000:4.0:1496184937.147027:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57303 00000100:00000040:4.0:1496184937.147029:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.147031:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.147038:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.147081:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184937.148632:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bba00 00000100:00020000:0.0:1496184937.159632:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bba00 00010000:00020000:4.0:1496184937.170684:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.170742:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.170747:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.170750:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.170755:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.170757:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.170769:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23731us (23831us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.170774:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57303 00000100:00000040:4.0:1496184937.170776:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.170779:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.170785:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.170788:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00000020:00000040:4.0:1496184937.170791:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.172463:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bddc50 x1568861104094576 msgsize 520 00000100:00100000:7.0:1496184937.172470:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.172471:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.172507:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094576 00000020:00000040:4.0:1496184937.172510:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.172511:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.172526:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43105 00000100:00000040:4.0:1496184937.172528:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.172530:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104094576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.172537:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094576:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.172543:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.172548:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.172551:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.172554:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.172557:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.172559:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.172560:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.172565:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.172570:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.172571:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.172572:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.172573:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.172575:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.172579:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.172580:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.172582:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.172584:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.172587:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.172589:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.172591:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.172592:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.172596:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094576 00000100:00000040:4.0:1496184937.172600:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.172603:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.172614:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.172617:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104094576/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.172623:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094576:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (159us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.172627:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43105 00000100:00000040:4.0:1496184937.172629:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.172632:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.173176:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.173179:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.173181:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.173186:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.173194:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.173203:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.173205:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00000100:00100000:4.0:1496184937.173215:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57304 00000100:00000040:4.0:1496184937.173217:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.173219:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.173225:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.173268:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.174860:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105ba000 00000100:00020000:3.0:1496184937.185865:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105ba000 00010000:00020000:4.0:1496184937.196933:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.196978:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.196983:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.196986:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.196991:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.196993:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.196999:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23774us (23873us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.197004:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57304 00000100:00000040:4.0:1496184937.197006:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.197010:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.197015:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.197020:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00000020:00000040:4.0:1496184937.197024:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.207522:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde450 x1568861104094608 msgsize 520 00000100:00100000:7.0:1496184937.207529:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.207530:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:4.0:1496184937.207566:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094608 00000020:00000040:4.0:1496184937.207569:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.207571:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.207586:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43106 00000100:00000040:4.0:1496184937.207587:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.207590:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104094608/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.207597:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094608:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.207603:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.207608:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.207611:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.207613:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.207617:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.207619:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.207620:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.207625:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.207628:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.207629:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.207630:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.207631:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.207633:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.207637:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.207638:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.207641:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.207642:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.207647:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.207650:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.207651:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.207653:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.207656:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094608 00000100:00000040:4.0:1496184937.207660:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.207663:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.207675:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.207677:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104094608/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.207683:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094608:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (160us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.207688:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43106 00000100:00000040:4.0:1496184937.207689:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.207692:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.208222:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.208225:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.208227:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.208232:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.208240:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.208246:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.208249:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00000100:00100000:4.0:1496184937.208258:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57305 00000100:00000040:4.0:1496184937.208260:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.208262:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.208269:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.208313:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.209666:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9e00 00000100:00020000:3.0:1496184937.220694:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9e00 00010000:00020000:4.0:1496184937.231754:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.231799:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.231804:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.231807:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.231812:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.231815:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.231821:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23552us (23649us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.231826:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57305 00000100:00000040:4.0:1496184937.231827:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.231831:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.231837:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.231839:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00000020:00000040:4.0:1496184937.231843:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.242448:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094640 00000020:00000040:4.0:1496184937.242451:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.242453:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.242467:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43107 00000100:00000040:4.0:1496184937.242469:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.242471:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104094640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.242479:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094640:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.242487:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.242492:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.242495:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.242497:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.242501:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.242503:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.242504:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.242509:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.242511:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.242513:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.242514:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.242515:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.242516:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.242520:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.242522:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.242524:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.242526:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.242528:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.242531:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.242533:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.242534:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.242537:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094640 00000100:00000040:4.0:1496184937.242542:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.242544:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.242556:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.242559:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104094640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.242565:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094640:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (166us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.242570:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43107 00000100:00000040:4.0:1496184937.242571:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.242574:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.243128:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.243131:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.243132:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.243138:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.243145:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.243152:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.243154:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00000100:00100000:4.0:1496184937.243164:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57306 00000100:00000040:4.0:1496184937.243166:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.243168:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.243175:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.243218:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.244635:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105b9000 00000100:00020000:3.0:1496184937.255661:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105b9000 00010000:00020000:4.0:1496184937.266751:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.266809:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.266814:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.266817:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.266822:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.266827:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.266833:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23658us (23748us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.266838:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57306 00000100:00000040:4.0:1496184937.266839:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.266843:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.266849:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.266851:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00000020:00000040:4.0:1496184937.266855:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.269744:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094672 00000020:00000040:4.0:1496184937.269747:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.269749:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:4.0:1496184937.269763:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43108 00000100:00000040:4.0:1496184937.269765:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.269767:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104094672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.269774:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094672:12345-10.37.248.124@o2ib1:8 00000020:00000040:4.0:1496184937.269780:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:4.0:1496184937.269785:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:4.0:1496184937.269788:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:4.0:1496184937.269791:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:4.0:1496184937.269798:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:4.0:1496184937.269799:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:4.0:1496184937.269801:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:4.0:1496184937.269808:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:4.0:1496184937.269810:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.269812:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.269813:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:4.0:1496184937.269813:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.269815:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.269819:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.269821:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:4.0:1496184937.269823:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.269825:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:4.0:1496184937.269827:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:4.0:1496184937.269830:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:4.0:1496184937.269831:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:4.0:1496184937.269833:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:4.0:1496184937.269836:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094672 00000100:00000040:4.0:1496184937.269840:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.269843:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.269859:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:4.0:1496184937.269862:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104094672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:4.0:1496184937.269868:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094672:12345-10.37.248.124@o2ib1:8 Request procesed in 93us (172us total) trans 0 rc 0/0 00000100:00100000:4.0:1496184937.269872:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43108 00000100:00000040:4.0:1496184937.269874:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:4.0:1496184937.269877:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:4.0:1496184937.270478:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:4.0:1496184937.270482:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:4.0:1496184937.270483:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:4.0:1496184937.270491:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:4.0:1496184937.270499:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:4.0:1496184937.270505:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.270508:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00000100:00100000:4.0:1496184937.270518:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57307 00000100:00000040:4.0:1496184937.270519:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:4.0:1496184937.270521:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/0 e 0 to 0 dl 1496184980 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:4.0:1496184937.270528:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:4.0:1496184937.270570:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184937.272416:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bac00 00000100:00020000:3.0:1496184937.283434:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bac00 00010000:00020000:4.0:1496184937.294490:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:4.0:1496184937.294548:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:4.0:1496184937.294553:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:4.0:1496184937.294556:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184937.294561:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:4.0:1496184937.294563:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184937.294569:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24041us (24139us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184937.294574:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57307 00000100:00000040:4.0:1496184937.294576:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:4.0:1496184937.294580:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:725/0 lens 608/448 e 0 to 0 dl 1496184980 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:4.0:1496184937.294592:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00002000:00010000:4.0:1496184937.294594:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00000020:00000040:4.0:1496184937.294598:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.304983:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdec50 x1568861104094704 msgsize 520 00000100:00100000:7.0:1496184937.304990:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.304991:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.305663:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.305670:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.305671:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.307395:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.307405:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00000040:7.0:1496184937.336393:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026072850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.336400:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.336401:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.337949:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.337960:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.369273:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.369299:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:2.0:1496184937.373851:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee6450 x1568861152397568 msgsize 224 00000100:00100000:2.0:1496184937.373859:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184937.373861:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184937.373912:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397568 00000020:00000040:1.0:1496184937.373917:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000040:1.0:1496184937.373920:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000100:00000040:1.0:1496184937.373931:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184937.373950:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 955 00000100:00000040:1.0:1496184937.373952:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000040:1.0:1496184937.373957:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee6450 x1568861152397568/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:688/0 lens 224/0 e 0 to 0 dl 1496184943 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184937.373966:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397568:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184937.373985:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 9758656 pending 0 free 80206888960 avail 76094111744 00010000:00000040:1.0:1496184937.373991:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397568 00000100:00000040:1.0:1496184937.373999:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184937.374004:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184937.374025:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184937.374027:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee6450 x1568861152397568/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:688/0 lens 224/336 e 0 to 0 dl 1496184943 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184937.374033:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397568:12345-10.37.248.197@o2ib1:13 Request procesed in 68us (182us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184937.374038:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 955 00000100:00000040:1.0:1496184937.374040:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000020:00000040:1.0:1496184937.374047:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 00000100:00020000:3.0:1496184937.400094:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.411242:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00000040:7.0:1496184937.432600:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c9050 x1568861104094832 msgsize 520 00000100:00100000:7.0:1496184937.432607:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.432609:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.433302:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.433309:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.433311:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.434915:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.445980:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00000040:1.0:1496184937.457121:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8cc50 x1568861104094864 msgsize 520 00000100:00100000:1.0:1496184937.457130:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184937.457132:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.457151:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094864 00000020:00000040:3.0:1496184937.457155:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184937.457159:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184937.457185:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6038 00000100:00000040:3.0:1496184937.457187:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.457193:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8cc50 x1568861104094864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184937.457201:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094864:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184937.457214:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.457226:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184937.457231:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.457234:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.457249:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.457253:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.457255:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.457264:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.457268:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.457270:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.457271:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184937.457272:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.457276:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.457281:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.457284:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.457287:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.457289:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.457294:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.457297:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.457301:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.457303:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.457312:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094864 00000100:00000040:3.0:1496184937.457318:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.457322:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.457341:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.457344:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8cc50 x1568861104094864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.457350:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094864:12345-10.37.248.124@o2ib1:8 Request procesed in 149us (228us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.457355:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6038 00000100:00000040:3.0:1496184937.457357:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.457363:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184937.470215:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.481211:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00000040:3.0:1496184937.502977:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b450 x1568861104094896 msgsize 520 00000100:00100000:3.0:1496184937.502985:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184937.502987:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184937.503030:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104094896 00000020:00000040:0.0:1496184937.503034:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184937.503037:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184937.503061:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6039 00000100:00000040:0.0:1496184937.503063:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184937.503066:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b450 x1568861104094896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184937.503074:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094896:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184937.503084:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184937.503092:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184937.503096:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184937.503098:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:0.0:1496184937.503104:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184937.503107:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184937.503108:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184937.503114:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184937.503118:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184937.503119:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184937.503120:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184937.503121:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184937.503123:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184937.503128:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184937.503130:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184937.503133:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184937.503135:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184937.503138:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184937.503141:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184937.503143:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184937.503144:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184937.503151:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104094896 00000100:00000040:0.0:1496184937.503156:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184937.503159:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184937.503177:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184937.503179:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b450 x1568861104094896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184937.503186:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104094896:12345-10.37.248.124@o2ib1:8 Request procesed in 112us (207us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184937.503190:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6039 00000100:00000040:0.0:1496184937.503192:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184937.503196:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.503727:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.503735:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.503736:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.505704:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.505728:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00000040:7.0:1496184937.534282:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8c50 x1568861104094928 msgsize 520 00000100:00100000:7.0:1496184937.534289:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.534290:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.535030:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026071050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.535037:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.535038:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.537083:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.537106:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:7.0:1496184937.565645:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8050 x1568861104094960 msgsize 520 00000100:00100000:7.0:1496184937.565652:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.565654:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.566369:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.566376:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.566377:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.568052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184937.568077:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184937.598656:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184937.609796:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00000040:7.0:1496184937.631676:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7c50 x1568861104095024 msgsize 520 00000100:00100000:7.0:1496184937.631683:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.631693:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.632390:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.632397:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.632400:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.634064:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184937.645141:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:0.0:1496184937.656270:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a050 x1568861104095056 msgsize 520 00000100:00100000:0.0:1496184937.656279:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184937.656281:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184937.656333:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095056 00000020:00000040:1.0:1496184937.656338:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:1.0:1496184937.656341:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:1.0:1496184937.656370:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6040 00000100:00000040:1.0:1496184937.656372:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:1.0:1496184937.656377:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a050 x1568861104095056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184937.656386:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095056:12345-10.37.248.124@o2ib1:8 00000020:00000040:1.0:1496184937.656410:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:1.0:1496184937.656422:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:1.0:1496184937.656427:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:1.0:1496184937.656430:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:1.0:1496184937.656439:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:1.0:1496184937.656442:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:1.0:1496184937.656445:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:1.0:1496184937.656453:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:1.0:1496184937.656457:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:1.0:1496184937.656459:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:1.0:1496184937.656460:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:1.0:1496184937.656461:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:1.0:1496184937.656465:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184937.656470:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:1.0:1496184937.656473:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:1.0:1496184937.656476:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184937.656478:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:1.0:1496184937.656484:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:1.0:1496184937.656487:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:1.0:1496184937.656489:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:1.0:1496184937.656491:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:1.0:1496184937.656499:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095056 00000100:00000040:1.0:1496184937.656506:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184937.656510:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:1.0:1496184937.656530:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:1.0:1496184937.656532:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a050 x1568861104095056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:1.0:1496184937.656548:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095056:12345-10.37.248.124@o2ib1:8 Request procesed in 163us (277us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184937.656553:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6040 00000100:00000040:1.0:1496184937.656555:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:1.0:1496184937.656560:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184937.669929:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.680927:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:1.0:1496184937.691977:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a850 x1568861104095088 msgsize 520 00000100:00100000:1.0:1496184937.691985:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184937.691987:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.692001:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095088 00000020:00000040:3.0:1496184937.692004:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184937.692007:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184937.692024:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6041 00000100:00000040:3.0:1496184937.692033:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184937.692037:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a850 x1568861104095088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184937.692045:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104095088:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184937.692052:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.692061:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 11 00010000:02000400:3.0:1496184937.692064:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.692067:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.692073:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.692075:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.692077:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.692082:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.692086:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000020:00000040:3.0:1496184937.692087:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000040:3.0:1496184937.692088:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000040:3.0:1496184937.692089:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184937.692092:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.692096:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.692098:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.692102:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.692104:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.692108:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.692110:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.692114:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000040:3.0:1496184937.692115:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.692121:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095088 00000100:00000040:3.0:1496184937.692126:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.692130:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.692144:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.692147:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a850 x1568861104095088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.692154:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095088:12345-10.37.248.124@o2ib1:8 Request procesed in 109us (175us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.692158:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6041 00000100:00000040:3.0:1496184937.692160:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.692164:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184937.694948:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184937.705951:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00020000:3.0:1496184937.730734:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184937.741780:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184937.763848:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7050 x1568861104095152 msgsize 520 00000100:00100000:7.0:1496184937.763856:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.763857:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.764611:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.764618:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.764628:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.766588:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184937.777616:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184937.799210:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7850 x1568861104095184 msgsize 520 00000100:00100000:7.0:1496184937.799217:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.799218:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.799948:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026009850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.799955:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.799961:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.801778:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.812815:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb4200 00000100:00020000:3.0:1496184937.836988:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184937.848029:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184937.859135:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a450 x1568861104095248 msgsize 520 00000100:00100000:1.0:1496184937.859144:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184937.859146:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.859167:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095248 00000020:00000040:3.0:1496184937.859170:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184937.859174:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184937.859200:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6042 00000100:00000040:3.0:1496184937.859202:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184937.859208:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a450 x1568861104095248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184937.859223:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095248:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184937.859236:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.859248:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184937.859253:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.859256:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.859272:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.859275:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.859278:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.859286:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.859290:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.859292:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.859293:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184937.859294:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.859297:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.859303:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.859306:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.859309:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.859311:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.859316:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.859319:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.859322:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.859323:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.859332:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095248 00000100:00000040:3.0:1496184937.859339:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.859343:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.859363:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.859366:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a450 x1568861104095248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.859372:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095248:12345-10.37.248.124@o2ib1:8 Request procesed in 150us (236us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.859377:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6042 00000100:00000040:3.0:1496184937.859380:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.859385:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184937.860316:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.860323:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.860325:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.870112:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184937.881125:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184937.902951:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6050 x1568861104095280 msgsize 520 00000100:00100000:7.0:1496184937.902958:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.902959:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184937.903613:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184937.903620:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184937.903622:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184937.905357:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.905368:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:0.0:1496184937.934034:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b850 x1568861104095312 msgsize 520 00000100:00100000:0.0:1496184937.934042:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184937.934044:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.934089:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095312 00000020:00000040:3.0:1496184937.934093:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184937.934096:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184937.934120:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6043 00000100:00000040:3.0:1496184937.934122:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.934127:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b850 x1568861104095312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184937.934135:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095312:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184937.934146:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.934156:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184937.934160:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.934163:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.934170:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.934173:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.934174:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.934185:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.934189:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.934191:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.934191:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184937.934192:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.934195:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.934200:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.934203:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.934206:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.934208:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.934212:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.934215:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.934217:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.934218:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.934225:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095312 00000100:00000040:3.0:1496184937.934231:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.934235:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.934253:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.934256:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b850 x1568861104095312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.934262:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095312:12345-10.37.248.124@o2ib1:8 Request procesed in 128us (227us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.934267:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6043 00000100:00000040:3.0:1496184937.934269:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.934274:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184937.936423:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.936450:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000040:0.0:1496184937.964818:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b050 x1568861104095344 msgsize 520 00000100:00100000:0.0:1496184937.964826:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184937.964827:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184937.964866:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095344 00000020:00000040:3.0:1496184937.964870:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184937.964872:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184937.964887:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6044 00000100:00000040:3.0:1496184937.964889:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.964899:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b050 x1568861104095344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184957 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184937.964907:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095344:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184937.964913:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184937.964919:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184937.964922:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184937.964925:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184937 last 1496184937 00000001:00000040:3.0:1496184937.964929:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184937.964931:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184937.964932:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184937.964938:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184937.964941:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.964942:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.964943:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184937.964944:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184937.964946:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.964950:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.964952:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184937.964954:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.964956:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184937.964959:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184937.964961:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184937.964965:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184937.964966:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184937.964971:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095344 00000100:00000040:3.0:1496184937.964975:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.964978:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.964991:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184937.964993:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b050 x1568861104095344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184957 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184937.964999:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095344:12345-10.37.248.124@o2ib1:8 Request procesed in 93us (180us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184937.965004:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6044 00000100:00000040:3.0:1496184937.965005:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184937.965009:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184937.967339:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.967364:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00020000:3.0:1496184937.999395:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.010557:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184938.032251:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5850 x1568861104095408 msgsize 520 00000100:00100000:7.0:1496184938.032258:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.032259:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.032932:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.032939:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.032940:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.034574:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184938.045619:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:1.0:1496184938.056756:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89450 x1568861104095440 msgsize 520 00000100:00100000:1.0:1496184938.056765:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184938.056767:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.056786:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095440 00000020:00000040:3.0:1496184938.056790:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.056794:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184938.056819:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6045 00000100:00000040:3.0:1496184938.056822:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.056827:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89450 x1568861104095440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.056836:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095440:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.056848:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.056861:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.056868:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.056872:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.056887:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.056897:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.056900:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.056908:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.056913:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.056915:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.056916:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.056917:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.056920:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.056926:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.056928:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.056932:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.056934:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.056939:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.056942:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.056944:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.056946:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.056954:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095440 00000100:00000040:3.0:1496184938.056961:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.056965:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.056985:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.056988:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89450 x1568861104095440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.056994:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095440:12345-10.37.248.124@o2ib1:8 Request procesed in 159us (237us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.057000:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6045 00000100:00000040:3.0:1496184938.057002:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.057008:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184938.059822:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184938.070817:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:7.0:1496184938.092808:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600cc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.092815:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.092823:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.094496:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184938.105527:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:7.0:1496184938.127623:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8450 x1568861104095504 msgsize 520 00000100:00100000:7.0:1496184938.127630:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.127631:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.128393:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.128400:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.128401:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.130498:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184938.141576:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:7.0:1496184938.164364:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.164371:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.164372:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.166171:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.177213:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.201589:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184938.212620:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184938.235055:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.235062:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.235063:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.236658:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.247691:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:1.0:1496184938.258794:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ac50 x1568861104095648 msgsize 520 00000100:00100000:1.0:1496184938.258803:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184938.258804:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.258824:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095648 00000020:00000040:3.0:1496184938.258828:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.258832:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184938.258857:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6046 00000100:00000040:3.0:1496184938.258860:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184938.258865:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ac50 x1568861104095648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.258874:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104095648:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.258887:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.258905:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.258910:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.258914:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.258922:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.258928:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.258937:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.258945:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.258950:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.258952:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.258953:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.258954:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.258957:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.258963:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.258965:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.258969:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.258971:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.258976:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.258979:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.258982:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.258983:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.258992:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095648 00000100:00000040:3.0:1496184938.258999:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.259003:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.259023:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.259026:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ac50 x1568861104095648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.259032:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095648:12345-10.37.248.124@o2ib1:8 Request procesed in 158us (237us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.259037:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6046 00000100:00000040:3.0:1496184938.259039:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.259046:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184938.259962:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600dc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.259969:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.259971:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.270154:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.281168:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.305275:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.305301:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184938.333984:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4c50 x1568861104095712 msgsize 520 00000100:00100000:7.0:1496184938.333991:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.333992:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.334688:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ec50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.334695:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.334696:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.336323:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184938.336348:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:0.0:1496184938.364538:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88850 x1568861104095744 msgsize 520 00000100:00100000:0.0:1496184938.364547:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184938.364548:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.364592:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095744 00000020:00000040:3.0:1496184938.364597:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.364601:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184938.364625:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6047 00000100:00000040:3.0:1496184938.364627:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.364632:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88850 x1568861104095744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.364640:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095744:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.364651:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.364662:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.364666:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.364669:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.364677:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.364680:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.364682:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.364689:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.364693:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.364695:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.364697:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.364698:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.364700:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.364708:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.364710:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.364713:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.364716:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.364720:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.364723:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.364725:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.364726:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.364734:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095744 00000100:00000040:3.0:1496184938.364741:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.364744:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.364762:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.364764:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88850 x1568861104095744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.364770:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095744:12345-10.37.248.124@o2ib1:8 Request procesed in 130us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.364775:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6047 00000100:00000040:3.0:1496184938.364777:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.364782:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184938.366831:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184938.366857:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184938.395211:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4050 x1568861104095776 msgsize 520 00000100:00100000:7.0:1496184938.395225:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.395226:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.397633:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.408798:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184938.430303:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4450 x1568861104095808 msgsize 520 00000100:00100000:7.0:1496184938.430310:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.430311:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.432789:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184938.443845:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:0.0:1496184938.454970:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89c50 x1568861104095840 msgsize 520 00000100:00100000:0.0:1496184938.454979:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184938.454980:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.455001:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104095840 00000020:00000040:3.0:1496184938.455005:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.455008:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184938.455034:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6048 00000100:00000040:3.0:1496184938.455039:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184938.455044:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89c50 x1568861104095840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.455053:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104095840:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.455066:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.455078:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.455083:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.455086:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.455095:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.455098:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.455100:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.455108:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.455120:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.455122:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.455123:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.455124:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.455127:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.455133:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.455136:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.455139:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.455141:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.455147:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.455149:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.455152:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.455154:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.455162:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104095840 00000100:00000040:3.0:1496184938.455171:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.455175:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.455195:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.455198:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89c50 x1568861104095840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.455204:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104095840:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (234us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.455210:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6048 00000100:00000040:3.0:1496184938.455212:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.455217:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184938.456165:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026078c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.456172:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.456174:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.457734:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184938.468755:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184938.495338:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184938.506370:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:7.0:1496184938.527792:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3c50 x1568861104095904 msgsize 520 00000100:00100000:7.0:1496184938.527799:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.527800:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.528535:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026077c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.528550:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.528551:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.530476:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184938.541535:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:7.0:1496184938.562848:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3850 x1568861104095936 msgsize 520 00000100:00100000:7.0:1496184938.562855:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.562857:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.565219:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.576275:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184938.597814:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3450 x1568861104095968 msgsize 520 00000100:00100000:7.0:1496184938.597821:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.597822:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.598568:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.598575:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.598576:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.600195:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184938.611233:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184938.634854:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184938.645902:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:1.0:1496184938.656992:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89850 x1568861104096032 msgsize 520 00000100:00100000:1.0:1496184938.657001:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184938.657010:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.657031:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104096032 00000020:00000040:3.0:1496184938.657035:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.657038:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184938.657063:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6049 00000100:00000040:3.0:1496184938.657065:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184938.657070:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89850 x1568861104096032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.657079:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104096032:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.657094:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.657106:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.657111:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.657115:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.657124:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.657127:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.657129:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.657137:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.657141:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.657143:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.657144:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.657152:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.657155:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.657161:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.657164:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.657167:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.657169:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.657175:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.657178:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.657180:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.657181:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.657190:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104096032 00000100:00000040:3.0:1496184938.657197:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.657201:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.657221:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.657224:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89850 x1568861104096032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.657231:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096032:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (238us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.657236:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6049 00000100:00000040:3.0:1496184938.657238:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.657244:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184938.658147:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026079850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.658154:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.658156:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.670145:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184938.681141:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184938.702921:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2c50 x1568861104096064 msgsize 520 00000100:00100000:7.0:1496184938.702928:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.702929:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.703598:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026077050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.703605:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.703606:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.705674:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184938.705699:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184938.733712:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2850 x1568861104096096 msgsize 520 00000100:00100000:7.0:1496184938.733719:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.733720:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.734434:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026078450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.734441:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.734443:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.735998:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.736021:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184938.766561:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184938.766585:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184938.796676:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184938.807820:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184938.831787:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184938.842861:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:0.0:1496184938.853971:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88050 x1568861104096224 msgsize 520 00000100:00100000:0.0:1496184938.853988:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184938.853990:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184938.854013:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104096224 00000020:00000040:3.0:1496184938.854017:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184938.854020:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184938.854045:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6050 00000100:00000040:3.0:1496184938.854047:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184938.854053:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88050 x1568861104096224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184958 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184938.854062:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104096224:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184938.854074:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184938.854087:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184938.854091:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184938.854097:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184938 last 1496184937 00000001:00000040:3.0:1496184938.854106:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184938.854109:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184938.854111:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184938.854119:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184938.854124:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.854126:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.854127:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184938.854135:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184938.854139:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.854144:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.854147:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184938.854150:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.854152:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184938.854157:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184938.854160:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184938.854163:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184938.854165:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184938.854173:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104096224 00000100:00000040:3.0:1496184938.854180:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.854184:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.854204:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184938.854207:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88050 x1568861104096224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184958 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184938.854213:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096224:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (242us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184938.854221:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6050 00000100:00000040:3.0:1496184938.854223:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184938.854229:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184938.855128:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.855136:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.855137:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.856552:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184938.867532:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:7.0:1496184938.893798:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0450 x1568861104096256 msgsize 520 00000100:00100000:7.0:1496184938.893805:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.893806:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.894495:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.894502:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.894506:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.896227:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184938.896252:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:7.0:1496184938.925536:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1450 x1568861104096288 msgsize 520 00000100:00100000:7.0:1496184938.925543:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.925544:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184938.926264:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026074450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184938.926271:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184938.926272:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184938.927766:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184938.927782:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184938.958775:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184938.958786:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184938.989064:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184938.989075:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184939.019390:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184939.030535:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184939.052347:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd8c50 x1568861104096416 msgsize 520 00000100:00100000:7.0:1496184939.052354:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.052356:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.054914:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184939.065972:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:7.0:1496184939.088395:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.088402:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.088403:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.089974:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184939.101052:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184939.123045:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104096480 msgsize 520 00000100:00100000:7.0:1496184939.123052:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.123053:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.123770:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.123776:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.123778:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.125756:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184939.136812:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184939.158235:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9050 x1568861104096512 msgsize 520 00000100:00100000:7.0:1496184939.158248:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.158249:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.160926:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184939.171964:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184939.196481:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184939.207528:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:7.0:1496184939.224858:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9850 x1568861104096576 msgsize 520 00000100:00100000:7.0:1496184939.224865:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.224866:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.225645:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026074c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.225659:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.225660:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.227240:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184939.238262:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:7.0:1496184939.260592:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.260606:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.260608:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.262068:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184939.273118:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184939.297527:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184939.297540:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:7.0:1496184939.325148:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdac50 x1568861104096672 msgsize 520 00000100:00100000:7.0:1496184939.325155:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.325157:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.325865:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026071c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.325872:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.325873:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.327408:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184939.327432:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:2.0:1496184939.341909:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184939.341922:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184939.341959:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.341971:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76a700] to pc [ptlrpcd_00_01:1] req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.341979:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184939.341983:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184939.341986:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.341992:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f769500] to pc [ptlrpcd_00_02:2] req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.341999:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184939.342001:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184939.342004:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342009:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76aa00] to pc [ptlrpcd_00_03:3] req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342016:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 4999 (4297378369) 00000100:00000040:0.0:1496184939.342021:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184939.342032:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184939.342034:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:d86c02e9-1939-02b8-4331-d3612fcf0106:13690:1568861159710192:10.37.248.196@o2ib1:400 00000100:00000040:3.0:1496184939.342035:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184939.342042:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:13691:1568861159710208:10.37.248.197@o2ib1:400 00000100:00100000:3.0:1496184939.342044:0:13692:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:13692:1568861159710224:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184939.342054:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184939.342056:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184939.342057:0:13692:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184939.342061:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:1.0:1496184939.342062:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184939.342063:0:13692:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:2.0:1496184939.342112:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184939.342121:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342127:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342391:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342419:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184939.342430:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184939.342458:0:13692:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:1.0:1496184939.342458:0:13691:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000040:2.0:1496184939.342461:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184939.342464:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:0.0:1496184939.342470:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184939.342472:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184939.342474:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00100000:2.0:1496184939.342477:0:13692:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:lustre-MDT0000-lwp-OST0001_UUID:13692:1568861159710224:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184939.342478:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184939.342480:0:13692:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76aa00 x1568861159710224/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00100000:1.0:1496184939.342480:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0000_UUID:13691:1568861159710208:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184939.342483:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f769500 x1568861159710208/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00100000:0.0:1496184939.342484:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:d86c02e9-1939-02b8-4331-d3612fcf0106:13690:1568861159710192:10.37.248.196@o2ib1:400 00000020:00000040:2.0:1496184939.342487:0:13692:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:0.0:1496184939.342487:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76a700 x1568861159710192/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184946 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:1.0:1496184939.342489:0:13691:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000020:00000040:0.0:1496184939.342493:0:13690:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000100:00000040:7.0:1496184939.355023:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb050 x1568861104096704 msgsize 520 00000100:00100000:7.0:1496184939.355039:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.355040:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.355742:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.355749:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.355751:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.357413:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184939.357438:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184939.384814:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104096736 msgsize 520 00000100:00100000:7.0:1496184939.384821:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.384822:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.385484:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026072450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.385491:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.385493:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.387335:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184939.387359:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184939.415262:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb850 x1568861104096768 msgsize 520 00000100:00100000:7.0:1496184939.415269:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.415271:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.415965:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.415972:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.415973:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.417546:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184939.428690:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:7.0:1496184939.449822:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb450 x1568861104096800 msgsize 520 00000100:00100000:7.0:1496184939.449829:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.449830:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.450515:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026072850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.450522:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.450530:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.452030:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184939.463085:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184939.484504:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc050 x1568861104096832 msgsize 520 00000100:00100000:7.0:1496184939.484511:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.484517:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.487433:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184939.498504:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184939.520266:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc450 x1568861104096864 msgsize 520 00000100:00100000:7.0:1496184939.520273:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.520274:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184939.521001:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026071450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184939.521008:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184939.521016:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184939.523038:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184939.534097:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00100000:7.0:1496184939.556425:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.556429:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.556430:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.556436:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.556448:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.556455:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.556457:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00000100:00100000:7.0:1496184939.556466:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57376 00000100:00000040:7.0:1496184939.556467:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.556470:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.556476:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.556521:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.558083:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184939.569152:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184939.580217:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.580275:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.580279:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.580284:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.580289:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.580291:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.580297:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23821us (23890us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.580302:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57376 00000100:00000040:7.0:1496184939.580304:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.580307:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.580313:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.580315:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00000020:00000040:7.0:1496184939.580319:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.590749:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104096928 00000020:00000040:7.0:1496184939.590752:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.590754:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.590769:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43165 00000100:00000040:7.0:1496184939.590770:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.590773:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdcc50 x1568861104096928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.590780:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096928:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.590786:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.590791:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.590794:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.590797:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.590801:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.590802:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.590804:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.590808:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.590811:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.590812:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.590813:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.590814:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.590816:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.590820:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.590822:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.590824:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.590826:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.590828:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.590830:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.590832:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.590833:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.590837:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104096928 00000100:00000040:7.0:1496184939.590841:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.590844:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.590855:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.590858:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdcc50 x1568861104096928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.590864:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096928:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.590868:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43165 00000100:00000040:7.0:1496184939.590870:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.590873:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.591467:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.591471:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.591472:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.591477:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.591485:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.591492:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.591494:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00000100:00100000:7.0:1496184939.591504:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57377 00000100:00000040:7.0:1496184939.591505:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.591508:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.591514:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.591557:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.593053:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184939.604109:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184939.615176:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.615222:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.615226:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.615229:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.615234:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.615237:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.615243:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23728us (23829us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.615248:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57377 00000100:00000040:7.0:1496184939.615250:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.615253:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.615259:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.615262:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00000020:00000040:7.0:1496184939.615265:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.625977:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104096976 00000020:00000040:7.0:1496184939.625980:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.625982:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.625996:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43166 00000100:00000040:7.0:1496184939.625998:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.626001:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd050 x1568861104096976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.626008:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096976:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.626014:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.626019:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.626022:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.626024:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.626028:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.626030:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.626031:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.626036:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.626038:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.626040:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.626041:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.626042:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.626044:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.626048:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.626049:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.626052:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.626054:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.626056:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.626058:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.626060:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.626061:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.626065:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104096976 00000100:00000040:7.0:1496184939.626069:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.626071:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.626083:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.626085:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd050 x1568861104096976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.626092:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104096976:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (158us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.626096:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43166 00000100:00000040:7.0:1496184939.626098:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.626101:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.626715:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.626718:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.626719:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.626725:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.626732:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.626739:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.626741:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00000100:00100000:7.0:1496184939.626751:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57378 00000100:00000040:7.0:1496184939.626753:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.626755:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.626762:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.626805:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.628479:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184939.639511:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184939.650575:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.650634:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.650638:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.650641:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.650646:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.650649:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.650655:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23893us (23992us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.650660:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57378 00000100:00000040:7.0:1496184939.650661:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.650665:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.650671:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.650673:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00000020:00000040:7.0:1496184939.650676:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.660951:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097008 00000020:00000040:7.0:1496184939.660954:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.660955:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.660970:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43167 00000100:00000040:7.0:1496184939.660972:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.660975:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd450 x1568861104097008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.660982:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097008:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.660988:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.660993:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.660996:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.660998:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.661002:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.661004:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.661005:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.661009:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.661012:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.661014:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.661014:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.661015:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.661017:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.661021:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.661023:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.661025:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.661027:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.661029:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.661031:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.661033:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.661034:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.661038:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097008 00000100:00000040:7.0:1496184939.661042:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.661045:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.661057:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.661059:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd450 x1568861104097008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.661065:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097008:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (179us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.661070:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43167 00000100:00000040:7.0:1496184939.661071:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.661074:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.661641:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.661644:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.661646:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.661651:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.661659:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.661665:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.661668:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00000100:00100000:7.0:1496184939.661678:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57379 00000100:00000040:7.0:1496184939.661679:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.661682:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.661689:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.661732:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.663402:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184939.674447:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184939.685521:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.685570:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.685575:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.685578:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.685583:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.685585:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.685591:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23902us (23996us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.685596:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57379 00000100:00000040:7.0:1496184939.685598:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.685601:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.685607:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.685609:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00000020:00000040:7.0:1496184939.685613:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.696143:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097040 00000020:00000040:7.0:1496184939.696146:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.696148:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.696163:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43168 00000100:00000040:7.0:1496184939.696165:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.696167:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd850 x1568861104097040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.696175:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097040:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.696180:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.696186:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.696189:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.696191:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.696195:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.696196:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.696198:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.696202:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.696205:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.696206:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.696207:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.696208:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.696210:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.696214:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.696216:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.696218:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.696220:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.696223:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.696225:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.696227:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.696228:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.696231:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097040 00000100:00000040:7.0:1496184939.696236:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.696238:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.696252:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.696254:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd850 x1568861104097040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.696261:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097040:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.696265:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43168 00000100:00000040:7.0:1496184939.696267:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.696270:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.696826:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.696829:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.696831:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.696837:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.696845:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.696851:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.696854:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00000100:00100000:7.0:1496184939.696864:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57380 00000100:00000040:7.0:1496184939.696866:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.696868:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.696875:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.696936:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.698543:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184939.698567:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00010000:00020000:7.0:1496184939.698609:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.698654:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.698659:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.698662:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.698667:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.698670:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.698680:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1805us (1898us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.698685:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57380 00000100:00000040:7.0:1496184939.698686:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.698690:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.698696:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.698698:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00000020:00000040:7.0:1496184939.698702:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.727570:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097072 00000020:00000040:7.0:1496184939.727573:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.727575:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.727589:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43169 00000100:00000040:7.0:1496184939.727591:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.727594:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104097072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.727601:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097072:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.727607:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.727612:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.727614:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.727617:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.727621:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.727622:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.727624:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.727628:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.727631:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.727633:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.727633:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.727634:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.727636:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.727640:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.727642:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.727644:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.727646:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.727649:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.727651:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.727653:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.727654:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.727657:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097072 00000100:00000040:7.0:1496184939.727662:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.727664:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.727676:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.727679:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104097072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.727685:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097072:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.727689:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43169 00000100:00000040:7.0:1496184939.727691:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.727694:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.728259:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.728262:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.728264:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.728269:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.728277:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.728284:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.728286:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000100:00100000:7.0:1496184939.728296:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57381 00000100:00000040:7.0:1496184939.728298:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.728300:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.728307:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.728348:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184939.729946:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:0.0:1496184939.729968:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184939.730013:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.730058:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.730062:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.730065:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.730070:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.730073:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.730079:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1772us (1862us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.730084:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57381 00000100:00000040:7.0:1496184939.730085:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.730089:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.730095:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.730097:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00000020:00000040:7.0:1496184939.730101:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.758452:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097104 00000020:00000040:7.0:1496184939.758455:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.758457:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.758471:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43170 00000100:00000040:7.0:1496184939.758472:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.758475:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104097104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.758482:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097104:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.758488:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.758493:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.758496:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.758498:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.758502:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.758504:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.758505:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.758510:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.758512:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.758514:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.758515:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.758515:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.758517:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.758521:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.758523:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.758525:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.758527:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.758530:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.758533:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.758534:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.758535:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.758539:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097104 00000100:00000040:7.0:1496184939.758543:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.758546:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.758558:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.758560:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104097104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.758566:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097104:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.758571:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43170 00000100:00000040:7.0:1496184939.758572:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.758575:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.759137:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.759140:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.759142:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.759147:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.759155:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.759161:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.759164:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000100:00100000:7.0:1496184939.759173:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57382 00000100:00000040:7.0:1496184939.759175:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.759177:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.759184:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.759228:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:0.0:1496184939.760975:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e400 00000100:00020000:0.0:1496184939.760998:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e400 00010000:00020000:7.0:1496184939.761041:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.761095:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.761099:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.761103:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.761108:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.761110:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.761116:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1932us (2021us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.761121:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57382 00000100:00000040:7.0:1496184939.761123:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.761126:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.761132:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.761134:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00000020:00000040:7.0:1496184939.761138:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.789316:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097136 00000020:00000040:7.0:1496184939.789319:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.789321:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.789335:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43171 00000100:00000040:7.0:1496184939.789337:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.789340:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104097136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.789347:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097136:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.789353:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.789358:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.789361:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.789364:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.789367:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.789369:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.789370:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.789375:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.789378:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.789379:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.789380:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.789381:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.789383:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.789387:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.789388:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.789391:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.789393:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.789395:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.789397:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.789399:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.789400:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.789404:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097136 00000100:00000040:7.0:1496184939.789408:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.789411:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.789423:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.789425:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104097136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.789431:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097136:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.789436:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43171 00000100:00000040:7.0:1496184939.789437:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.789440:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.790011:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.790014:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.790015:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.790020:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.790028:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.790035:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.790037:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000100:00100000:7.0:1496184939.790047:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57383 00000100:00000040:7.0:1496184939.790049:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.790051:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.790057:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.790101:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.791633:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184939.791660:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184939.791704:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.791749:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.791754:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.791757:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.791762:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.791764:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.791770:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1713us (1804us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.791775:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57383 00000100:00000040:7.0:1496184939.791777:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.791781:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.791786:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.791789:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00000020:00000040:7.0:1496184939.791792:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.820395:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097168 00000020:00000040:7.0:1496184939.820399:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.820400:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.820414:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43172 00000100:00000040:7.0:1496184939.820416:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.820419:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104097168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.820426:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097168:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.820432:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.820437:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.820440:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.820442:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.820446:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.820448:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.820449:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.820454:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.820457:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.820459:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.820460:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.820460:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.820462:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.820466:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.820468:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.820470:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.820472:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.820474:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.820477:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.820478:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.820480:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.820483:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097168 00000100:00000040:7.0:1496184939.820487:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.820490:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.820502:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.820504:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104097168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.820510:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097168:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.820515:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43172 00000100:00000040:7.0:1496184939.820516:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.820519:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.821159:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.821162:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.821164:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.821170:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.821178:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.821185:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.821187:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00000100:00100000:7.0:1496184939.821197:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57384 00000100:00000040:7.0:1496184939.821198:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.821201:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.821207:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.821250:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.822695:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184939.833858:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184939.844949:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.845006:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.845010:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.845013:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.845018:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.845021:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.845027:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23820us (23918us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.845032:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57384 00000100:00000040:7.0:1496184939.845033:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.845037:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.845047:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.845050:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00000020:00000040:7.0:1496184939.845053:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.856005:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097200 00000020:00000040:7.0:1496184939.856010:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.856012:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.856034:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43173 00000100:00000040:7.0:1496184939.856036:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.856052:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdec50 x1568861104097200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.856059:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097200:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.856065:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.856070:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.856073:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.856075:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.856079:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.856081:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.856082:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.856086:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.856089:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.856090:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.856091:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.856092:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.856094:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.856098:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.856100:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.856102:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.856104:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.856107:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.856109:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.856111:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.856112:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.856115:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097200 00000100:00000040:7.0:1496184939.856119:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.856122:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.856134:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.856136:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdec50 x1568861104097200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.856142:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097200:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (208us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.856147:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43173 00000100:00000040:7.0:1496184939.856148:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.856151:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.856739:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.856742:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.856744:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.856749:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.856757:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.856764:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.856766:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00000100:00100000:7.0:1496184939.856776:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57385 00000100:00000040:7.0:1496184939.856777:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.856780:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.856787:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.856831:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.858332:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184939.869393:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00010000:00020000:7.0:1496184939.880461:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.880506:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.880511:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.880514:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.880519:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.880521:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.880527:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23740us (23841us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.880532:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57385 00000100:00000040:7.0:1496184939.880534:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.880537:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.880543:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.880546:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00000020:00000040:7.0:1496184939.880549:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.891224:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097232 00000020:00000040:7.0:1496184939.891228:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.891229:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.891244:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43174 00000100:00000040:7.0:1496184939.891245:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.891248:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf050 x1568861104097232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.891255:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097232:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.891261:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.891267:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.891269:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.891272:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.891275:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.891277:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.891279:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.891283:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.891286:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.891287:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.891288:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.891289:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.891291:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.891295:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.891296:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.891298:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.891300:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.891303:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.891305:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.891307:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.891308:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.891312:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097232 00000100:00000040:7.0:1496184939.891316:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.891319:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.891330:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.891333:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf050 x1568861104097232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.891339:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097232:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.891343:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43174 00000100:00000040:7.0:1496184939.891345:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.891348:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.891940:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.891943:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.891952:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.891957:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.891965:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.891972:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.891974:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00000100:00100000:7.0:1496184939.891984:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57386 00000100:00000040:7.0:1496184939.891986:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.891988:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.891994:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.892041:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.893672:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184939.904769:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184939.915847:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.915905:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.915910:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.915913:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.915919:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.915921:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.915927:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23933us (24047us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.915932:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57386 00000100:00000040:7.0:1496184939.915934:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.915937:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.915943:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.915946:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00000020:00000040:7.0:1496184939.915949:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.926328:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097264 00000020:00000040:7.0:1496184939.926331:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.926333:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.926347:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43175 00000100:00000040:7.0:1496184939.926349:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.926352:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf450 x1568861104097264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.926359:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097264:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.926365:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.926370:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.926373:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.926375:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.926379:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.926381:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.926383:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.926387:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.926390:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.926391:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.926392:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.926393:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.926395:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.926399:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.926400:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.926403:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.926405:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.926407:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.926409:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.926411:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.926412:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.926416:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097264 00000100:00000040:7.0:1496184939.926420:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.926423:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.926435:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.926437:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf450 x1568861104097264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.926443:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097264:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.926448:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43175 00000100:00000040:7.0:1496184939.926449:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.926452:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.927000:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.927003:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.927005:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.927011:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.927018:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.927025:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.927027:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00000100:00100000:7.0:1496184939.927037:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57387 00000100:00000040:7.0:1496184939.927039:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.927041:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.927047:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.927091:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.928784:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184939.939847:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184939.950929:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.950975:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.950979:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.950982:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.950988:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.950990:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.950996:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23949us (24046us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.951001:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57387 00000100:00000040:7.0:1496184939.951003:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.951007:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.951012:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.951015:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00000020:00000040:7.0:1496184939.951018:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.961267:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097296 00000020:00000040:7.0:1496184939.961271:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.961272:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.961286:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43176 00000100:00000040:7.0:1496184939.961288:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.961291:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf850 x1568861104097296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.961298:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097296:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.961304:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.961309:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.961312:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.961315:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.961319:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.961320:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.961322:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.961326:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.961329:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.961330:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.961331:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.961332:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.961334:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.961339:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.961340:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.961342:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.961344:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.961347:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.961349:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.961351:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.961352:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.961356:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097296 00000100:00000040:7.0:1496184939.961360:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.961362:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.961374:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.961377:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf850 x1568861104097296/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.961383:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097296:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (166us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.961387:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43176 00000100:00000040:7.0:1496184939.961389:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.961392:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.961982:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.961985:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.961986:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.961991:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.961999:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.962006:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.962008:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000100:00100000:7.0:1496184939.962018:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57388 00000100:00000040:7.0:1496184939.962020:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.962023:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.962029:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.962073:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.963515:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e600 00000100:00020000:3.0:1496184939.974550:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e600 00010000:00020000:7.0:1496184939.985619:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184939.985677:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184939.985682:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184939.985685:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184939.985690:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184939.985693:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184939.985699:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23669us (23762us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184939.985704:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57388 00000100:00000040:7.0:1496184939.985705:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184939.985709:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184939.985714:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.985717:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00000020:00000040:7.0:1496184939.985720:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.996023:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097328 00000020:00000040:7.0:1496184939.996026:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.996027:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184939.996042:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43177 00000100:00000040:7.0:1496184939.996044:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.996047:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c9050 x1568861104097328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184959 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.996054:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097328:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184939.996060:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184939.996065:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184939.996068:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184939.996070:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184939 last 1496184939 00000001:00000040:7.0:1496184939.996074:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184939.996076:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184939.996077:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184939.996081:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184939.996084:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.996085:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.996086:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184939.996087:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.996089:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.996093:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.996094:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184939.996097:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.996099:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184939.996101:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184939.996103:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184939.996105:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184939.996106:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184939.996110:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097328 00000100:00000040:7.0:1496184939.996114:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.996117:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.996128:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184939.996131:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c9050 x1568861104097328/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184959 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184939.996137:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097328:12345-10.37.248.124@o2ib1:8 Request procesed in 83us (157us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184939.996141:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43177 00000100:00000040:7.0:1496184939.996143:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184939.996146:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184939.996693:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184939.996696:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184939.996698:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184939.996704:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184939.996712:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184939.996718:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184939.996721:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000100:00100000:7.0:1496184939.996730:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57389 00000100:00000040:7.0:1496184939.996732:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184939.996734:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/0 e 0 to 0 dl 1496184982 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184939.996741:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184939.996782:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184939.998318:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f800 00000100:00020000:3.0:1496184940.009365:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f800 00010000:00020000:7.0:1496184940.020420:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.020478:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.020483:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.020486:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.020491:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.020494:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.020500:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23759us (23850us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.020505:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57389 00000100:00000040:7.0:1496184940.020506:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.020510:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:727/0 lens 608/448 e 0 to 0 dl 1496184982 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.020515:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.020518:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00000020:00000040:7.0:1496184940.020521:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.030604:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097360 00000020:00000040:7.0:1496184940.030607:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.030608:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.030623:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43178 00000100:00000040:7.0:1496184940.030625:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.030628:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8c50 x1568861104097360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.030635:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097360:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.030641:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.030646:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.030649:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.030651:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.030655:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.030657:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.030658:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.030663:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.030666:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.030667:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.030668:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.030669:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.030671:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.030675:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.030676:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.030678:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.030680:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.030683:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.030685:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.030687:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.030688:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.030691:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097360 00000100:00000040:7.0:1496184940.030696:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.030698:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.030710:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.030713:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8c50 x1568861104097360/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.030719:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097360:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.030723:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43178 00000100:00000040:7.0:1496184940.030725:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.030729:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.031303:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.031307:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.031308:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.031314:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.031322:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.031328:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.031331:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:7.0:1496184940.031341:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57390 00000100:00000040:7.0:1496184940.031342:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.031344:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.031351:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.031394:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.033081:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184940.044138:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184940.055225:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184940.055248:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88450 x1568861104097392 msgsize 520 00000100:00100000:1.0:1496184940.055256:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184940.055258:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184940.055278:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097392 00000020:00000040:7.0:1496184940.055282:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.055282:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184940.055285:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:02000400:7.0:1496184940.055287:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.055290:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.055296:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.055298:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.055304:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23953us (24050us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.055310:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57390 00000100:00100000:3.0:1496184940.055310:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6051 00000100:00000040:7.0:1496184940.055312:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.055312:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00002000:00100000:7.0:1496184940.055316:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184940.055317:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88450 x1568861104097392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.055322:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.055325:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00000100:00100000:3.0:1496184940.055327:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104097392:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.055329:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.055339:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184940.055352:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184940.055356:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184940.055367:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:3.0:1496184940.055376:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184940.055387:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184940.055389:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184940.055397:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184940.055402:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.055404:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.055405:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184940.055406:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.055409:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.055415:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.055417:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184940.055420:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.055423:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.055428:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184940.055431:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184940.055434:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.055435:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184940.055444:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097392 00000100:00000040:3.0:1496184940.055451:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.055455:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.055475:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.055478:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88450 x1568861104097392/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184940.055484:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097392:12345-10.37.248.124@o2ib1:8 Request procesed in 158us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184940.055492:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6051 00000100:00000040:3.0:1496184940.055494:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.055500:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.056486:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.056490:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.056492:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.056497:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.056505:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.056511:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.056514:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000100:00100000:7.0:1496184940.056524:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57391 00000100:00000040:7.0:1496184940.056525:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.056528:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.056534:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.056578:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.058208:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184940.069230:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184940.080275:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.080333:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.080338:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.080341:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.080346:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.080349:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.080355:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23821us (23914us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.080360:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57391 00000100:00000040:7.0:1496184940.080362:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.080366:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.080371:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.080374:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00000020:00000040:7.0:1496184940.080377:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.091039:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097424 00000020:00000040:7.0:1496184940.091042:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.091044:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.091058:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43179 00000100:00000040:7.0:1496184940.091060:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.091062:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8050 x1568861104097424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.091070:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097424:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.091076:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.091081:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.091084:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.091086:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.091091:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.091092:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.091094:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.091099:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.091102:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.091103:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.091104:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.091105:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.091107:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.091111:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.091113:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.091115:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.091117:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.091120:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.091122:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.091124:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.091125:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.091129:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097424 00000100:00000040:7.0:1496184940.091133:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.091136:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.091147:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.091150:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8050 x1568861104097424/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.091156:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097424:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (170us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.091160:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43179 00000100:00000040:7.0:1496184940.091162:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.091165:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.091671:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.091674:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.091676:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.091682:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.091690:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.091697:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.091699:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000100:00100000:7.0:1496184940.091709:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57392 00000100:00000040:7.0:1496184940.091710:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.091713:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.091719:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.091761:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.093320:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184940.104350:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184940.115403:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.115461:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.115466:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.115469:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.115474:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.115477:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.115483:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23763us (23855us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.115488:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57392 00000100:00000040:7.0:1496184940.115489:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.115493:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.115504:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.115506:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00000020:00000040:7.0:1496184940.115509:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.126430:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097456 00000020:00000040:7.0:1496184940.126433:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.126435:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.126450:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43180 00000100:00000040:7.0:1496184940.126451:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.126454:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7450 x1568861104097456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.126461:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097456:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.126467:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.126472:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.126475:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.126478:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.126481:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.126483:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.126484:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.126489:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.126492:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.126493:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.126494:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.126495:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.126497:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.126501:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.126503:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.126505:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.126507:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.126510:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.126512:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.126514:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.126515:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.126518:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097456 00000100:00000040:7.0:1496184940.126523:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.126525:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.126537:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.126540:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7450 x1568861104097456/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.126546:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097456:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.126550:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43180 00000100:00000040:7.0:1496184940.126551:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.126555:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.127149:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.127152:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.127154:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.127159:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.127167:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.127173:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.127176:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000100:00100000:7.0:1496184940.127186:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57393 00000100:00000040:7.0:1496184940.127187:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.127190:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.127196:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.127239:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.129032:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184940.140092:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184940.151163:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.151221:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.151226:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.151229:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.151234:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.151237:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.151242:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24046us (24138us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.151247:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57393 00000100:00000040:7.0:1496184940.151249:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.151253:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.151258:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.151261:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00000020:00000040:7.0:1496184940.151264:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.161918:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097488 00000020:00000040:7.0:1496184940.161921:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.161923:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.161938:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43181 00000100:00000040:7.0:1496184940.161939:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.161942:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7c50 x1568861104097488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.161950:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097488:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.161955:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.161961:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.161963:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.161966:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.161970:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.161971:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.161973:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.161979:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.161982:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.161983:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.161984:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.161985:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.161987:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.161991:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.161993:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.161995:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.161997:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.162000:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.162002:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.162004:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.162005:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.162008:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097488 00000100:00000040:7.0:1496184940.162013:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.162015:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.162027:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.162030:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7c50 x1568861104097488/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.162036:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097488:12345-10.37.248.124@o2ib1:8 Request procesed in 86us (165us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.162040:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43181 00000100:00000040:7.0:1496184940.162042:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.162045:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.162633:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.162636:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.162637:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.162642:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.162650:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.162657:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.162659:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000100:00100000:7.0:1496184940.162669:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57394 00000100:00000040:7.0:1496184940.162671:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.162673:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.162680:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.162723:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.164487:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e000 00000100:00020000:3.0:1496184940.175537:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e000 00010000:00020000:7.0:1496184940.186608:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.186666:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.186671:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.186674:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.186679:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.186682:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.186688:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24008us (24109us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.186693:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57394 00000100:00000040:7.0:1496184940.186694:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.186698:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.186703:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.186706:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00000020:00000040:7.0:1496184940.186709:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.197372:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097520 00000020:00000040:7.0:1496184940.197375:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.197377:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.197391:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43182 00000100:00000040:7.0:1496184940.197393:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.197395:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6c50 x1568861104097520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.197403:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097520:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.197408:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.197413:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.197416:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.197419:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.197422:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.197425:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.197426:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.197430:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.197433:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.197434:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.197435:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.197436:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.197438:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.197442:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.197444:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.197446:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.197448:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.197450:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.197453:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.197454:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.197455:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.197459:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097520 00000100:00000040:7.0:1496184940.197463:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.197466:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.197478:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.197481:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6c50 x1568861104097520/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.197487:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097520:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.197491:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43182 00000100:00000040:7.0:1496184940.197493:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.197496:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.198079:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.198083:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.198084:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.198090:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.198098:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.198105:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.198107:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000100:00100000:7.0:1496184940.198117:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57395 00000100:00000040:7.0:1496184940.198119:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.198121:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.198127:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.198170:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.199720:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f400 00000100:00020000:3.0:1496184940.210761:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f400 00010000:00020000:7.0:1496184940.221824:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.221881:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.221885:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.221888:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.221910:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.221913:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.221919:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23791us (23883us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.221924:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57395 00000100:00000040:7.0:1496184940.221926:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.221930:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.221935:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.221938:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00000020:00000040:7.0:1496184940.221941:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.232705:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097552 00000020:00000040:7.0:1496184940.232708:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.232710:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.232724:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43183 00000100:00000040:7.0:1496184940.232726:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.232729:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7050 x1568861104097552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.232736:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097552:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.232742:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.232747:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.232750:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.232752:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.232756:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.232758:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.232759:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.232764:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.232766:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.232768:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.232769:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.232769:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.232771:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.232776:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.232777:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.232780:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.232782:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.232784:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.232786:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.232788:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.232789:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.232793:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097552 00000100:00000040:7.0:1496184940.232797:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.232800:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.232812:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.232814:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7050 x1568861104097552/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.232820:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097552:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (159us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.232825:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43183 00000100:00000040:7.0:1496184940.232826:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.232829:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.233368:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.233371:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.233372:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.233378:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.233386:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.233393:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.233395:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000100:00100000:7.0:1496184940.233405:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57396 00000100:00000040:7.0:1496184940.233407:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.233409:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.233416:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.233460:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.235170:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184940.246222:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00010000:00020000:7.0:1496184940.257305:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184940.257306:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88c50 x1568861104097584 msgsize 520 00000100:00100000:1.0:1496184940.257324:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184940.257325:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184940.257346:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097584 00000020:00000040:3.0:1496184940.257349:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184940.257353:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:7.0:1496184940.257364:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:7.0:1496184940.257369:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.257372:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.257377:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00100000:3.0:1496184940.257377:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6052 00000100:00000040:7.0:1496184940.257380:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184940.257380:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184940.257385:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88c50 x1568861104097584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.257386:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23970us (24063us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.257392:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57396 00000100:00000040:7.0:1496184940.257394:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00100000:3.0:1496184940.257394:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104097584:12345-10.37.248.124@o2ib1:8 00002000:00100000:7.0:1496184940.257397:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.257403:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.257406:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00000020:00000040:3.0:1496184940.257406:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.257410:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.257419:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184940.257423:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184940.257427:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:3.0:1496184940.257436:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184940.257441:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184940.257444:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184940.257459:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184940.257464:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.257466:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.257467:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184940.257468:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.257471:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.257477:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.257479:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184940.257483:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.257485:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.257490:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184940.257493:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184940.257496:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.257497:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184940.257506:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097584 00000100:00000040:3.0:1496184940.257513:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.257517:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.257536:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.257538:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88c50 x1568861104097584/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184940.257545:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097584:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (238us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184940.257550:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6052 00000100:00000040:3.0:1496184940.257552:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.257558:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.258543:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.258546:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.258548:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.258555:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.258562:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.258569:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.258572:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000100:00100000:7.0:1496184940.258581:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57397 00000100:00000040:7.0:1496184940.258583:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.258585:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.258592:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.258643:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.270196:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184940.281207:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00010000:00020000:7.0:1496184940.292256:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.292313:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.292318:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.292321:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.292326:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.292328:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.292334:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33742us (33836us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.292340:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57397 00000100:00000040:7.0:1496184940.292341:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.292345:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.292355:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.292358:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00000020:00000040:7.0:1496184940.292361:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.302904:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097616 00000020:00000040:7.0:1496184940.302907:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.302909:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.302923:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43184 00000100:00000040:7.0:1496184940.302925:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.302927:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7850 x1568861104097616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.302935:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097616:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.302941:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.302946:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.302949:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.302951:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.302955:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.302957:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.302959:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.302964:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.302967:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.302968:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.302969:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.302970:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.302972:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.302976:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.302978:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.302981:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.302983:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.302986:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.302988:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.302990:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.302991:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.302995:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097616 00000100:00000040:7.0:1496184940.302999:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.303001:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.303018:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.303021:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7850 x1568861104097616/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.303027:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097616:12345-10.37.248.124@o2ib1:8 Request procesed in 92us (179us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.303031:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43184 00000100:00000040:7.0:1496184940.303033:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.303036:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.303548:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.303551:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.303553:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.303558:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.303566:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.303573:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.303575:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000100:00100000:7.0:1496184940.303585:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57398 00000100:00000040:7.0:1496184940.303587:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.303589:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.303596:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.303640:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.305422:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184940.305448:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184940.305490:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.305535:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.305540:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.305543:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.305548:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.305551:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.305557:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1961us (2053us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.305562:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57398 00000100:00000040:7.0:1496184940.305563:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.305567:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.305573:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.305575:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00000020:00000040:7.0:1496184940.305578:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.333781:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097648 00000020:00000040:7.0:1496184940.333784:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.333785:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.333799:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43185 00000100:00000040:7.0:1496184940.333801:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.333804:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6850 x1568861104097648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.333811:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097648:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.333817:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.333822:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.333825:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.333827:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.333831:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.333833:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.333834:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.333839:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.333842:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.333843:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.333844:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.333845:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.333847:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.333851:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.333852:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.333854:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.333856:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.333859:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.333861:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.333863:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.333864:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.333867:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097648 00000100:00000040:7.0:1496184940.333872:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.333875:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.333887:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.333909:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6850 x1568861104097648/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.333915:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097648:12345-10.37.248.124@o2ib1:8 Request procesed in 104us (182us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.333919:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43185 00000100:00000040:7.0:1496184940.333921:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.333924:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.334384:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.334387:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.334389:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.334395:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.334403:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.334409:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.334412:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000100:00100000:7.0:1496184940.334422:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57399 00000100:00000040:7.0:1496184940.334423:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.334426:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.334432:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.334475:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.335983:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184940.336008:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184940.336053:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.336098:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.336103:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.336106:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.336111:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.336114:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.336120:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1687us (1780us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.336125:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57399 00000100:00000040:7.0:1496184940.336126:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.336130:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.336136:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.336138:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00000020:00000040:7.0:1496184940.336142:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.364780:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097680 00000020:00000040:7.0:1496184940.364783:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.364784:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.364799:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43186 00000100:00000040:7.0:1496184940.364800:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.364803:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6050 x1568861104097680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.364810:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097680:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.364816:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.364822:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.364824:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.364827:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.364831:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.364832:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.364834:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.364838:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.364841:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.364842:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.364843:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.364844:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.364846:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.364850:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.364852:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.364854:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.364856:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.364858:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.364860:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.364862:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.364863:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.364867:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097680 00000100:00000040:7.0:1496184940.364871:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.364874:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.364886:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.364888:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6050 x1568861104097680/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.364905:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097680:12345-10.37.248.124@o2ib1:8 Request procesed in 94us (174us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.364909:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43186 00000100:00000040:7.0:1496184940.364911:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.364914:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.365442:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.365446:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.365447:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.365452:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.365460:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.365467:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.365469:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000100:00100000:7.0:1496184940.365479:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57400 00000100:00000040:7.0:1496184940.365480:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.365483:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.365489:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.365533:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.367183:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fe00 00000100:00020000:3.0:1496184940.367207:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fe00 00010000:00020000:7.0:1496184940.367249:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.367294:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.367299:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.367302:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.367307:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.367309:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.367315:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1826us (1919us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.367320:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57400 00000100:00000040:7.0:1496184940.367322:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.367326:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.367331:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.367334:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00000020:00000040:7.0:1496184940.367337:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.395338:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097712 00000020:00000040:7.0:1496184940.395341:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.395343:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.395357:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43187 00000100:00000040:7.0:1496184940.395359:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.395362:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5c50 x1568861104097712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.395369:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097712:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.395375:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.395380:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.395382:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.395385:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.395389:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.395391:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.395392:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.395396:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.395399:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.395400:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.395401:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.395402:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.395404:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.395409:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.395410:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.395412:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.395414:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.395417:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.395419:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.395421:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.395422:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.395426:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097712 00000100:00000040:7.0:1496184940.395430:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.395432:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.395444:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.395447:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5c50 x1568861104097712/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.395453:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097712:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.395457:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43187 00000100:00000040:7.0:1496184940.395459:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.395462:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.396019:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.396022:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.396023:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.396028:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.396036:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.396043:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.396045:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000100:00100000:7.0:1496184940.396055:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57401 00000100:00000040:7.0:1496184940.396057:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.396059:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.396066:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.396109:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.397688:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f600 00000100:00020000:3.0:1496184940.408843:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f600 00010000:00020000:7.0:1496184940.419919:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.419976:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.419980:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.419984:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.419989:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.419991:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.419997:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23931us (24032us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.420002:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57401 00000100:00000040:7.0:1496184940.420004:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.420007:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.420013:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.420015:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00000020:00000040:7.0:1496184940.420019:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.430211:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097744 00000020:00000040:7.0:1496184940.430215:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.430216:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.430231:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43188 00000100:00000040:7.0:1496184940.430232:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.430235:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5850 x1568861104097744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.430242:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097744:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.430248:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.430253:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.430256:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.430258:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.430262:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.430264:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.430265:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.430270:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.430273:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.430274:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.430275:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.430276:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.430278:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.430282:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.430283:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.430285:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.430287:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.430290:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.430292:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.430294:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.430295:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.430299:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097744 00000100:00000040:7.0:1496184940.430303:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.430306:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.430318:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.430321:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5850 x1568861104097744/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.430327:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097744:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.430332:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43188 00000100:00000040:7.0:1496184940.430333:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.430336:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.430909:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.430912:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.430914:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.430919:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.430927:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.430934:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.430936:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000100:00100000:7.0:1496184940.430946:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57402 00000100:00000040:7.0:1496184940.430947:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.430950:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.430956:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.431001:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.432484:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184940.443553:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00000100:00000040:1.0:1496184940.454691:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285050 x1568861104097776 msgsize 520 00010000:00020000:7.0:1496184940.454692:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00100000:1.0:1496184940.454700:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184940.454702:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184940.454721:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097776 00000020:00000040:3.0:1496184940.454725:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184940.454728:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:7.0:1496184940.454738:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:02000400:7.0:1496184940.454743:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.454746:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.454751:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00100000:3.0:1496184940.454752:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6053 00000100:00000040:7.0:1496184940.454754:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184940.454755:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00100000:7.0:1496184940.454760:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23804us (23911us total) trans 0 rc -110/-999 00000100:00000040:3.0:1496184940.454760:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285050 x1568861104097776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.454766:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57402 00000100:00000040:7.0:1496184940.454767:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00100000:3.0:1496184940.454769:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104097776:12345-10.37.248.124@o2ib1:8 00002000:00100000:7.0:1496184940.454772:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.454777:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.454780:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00000020:00000040:3.0:1496184940.454782:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.454784:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.454794:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184940.454799:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184940.454802:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:3.0:1496184940.454811:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184940.454813:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184940.454816:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184940.454824:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184940.454831:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.454840:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.454841:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184940.454842:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.454845:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.454851:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.454854:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184940.454857:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.454859:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.454865:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184940.454868:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184940.454870:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.454872:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184940.454880:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097776 00000100:00000040:3.0:1496184940.454887:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.454899:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.454918:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.454921:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285050 x1568861104097776/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184940.454927:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097776:12345-10.37.248.124@o2ib1:8 Request procesed in 159us (235us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184940.454933:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6053 00000100:00000040:3.0:1496184940.454935:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.454941:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.455887:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.455902:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.455904:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.455910:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.455918:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.455925:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.455927:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000100:00100000:7.0:1496184940.455937:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57403 00000100:00000040:7.0:1496184940.455938:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.455941:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.455952:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.455994:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.457425:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c800 00000100:00020000:3.0:1496184940.468432:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c800 00010000:00020000:7.0:1496184940.479483:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.479529:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.479533:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.479536:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.479542:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.479544:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.479550:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23598us (23709us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.479555:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57403 00000100:00000040:7.0:1496184940.479557:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.479561:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.479566:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.479569:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00000020:00000040:7.0:1496184940.479572:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.493368:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097808 00000020:00000040:7.0:1496184940.493371:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.493373:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.493387:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43189 00000100:00000040:7.0:1496184940.493389:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.493392:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6450 x1568861104097808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.493399:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097808:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.493405:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.493410:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.493413:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.493416:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.493420:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.493422:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.493423:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.493428:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.493431:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.493432:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.493433:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.493434:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.493436:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.493441:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.493442:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.493445:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.493447:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.493449:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.493452:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.493454:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.493455:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.493459:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097808 00000100:00000040:7.0:1496184940.493463:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.493465:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.493478:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.493480:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6450 x1568861104097808/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.493487:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097808:12345-10.37.248.124@o2ib1:8 Request procesed in 87us (167us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.493491:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43189 00000100:00000040:7.0:1496184940.493492:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.493496:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.494027:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.494030:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.494032:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.494037:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.494045:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.494052:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.494054:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000100:00100000:7.0:1496184940.494064:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57404 00000100:00000040:7.0:1496184940.494066:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.494068:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.494075:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.494119:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.495672:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ce00 00000100:00020000:3.0:1496184940.506701:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ce00 00010000:00020000:7.0:1496184940.517768:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.517813:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.517818:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.517821:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.517830:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.517833:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.517839:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23764us (23855us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.517844:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57404 00000100:00000040:7.0:1496184940.517845:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.517849:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.517854:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.517857:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000020:00000040:7.0:1496184940.517860:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.528520:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097840 00000020:00000040:7.0:1496184940.528523:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.528525:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.528540:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43190 00000100:00000040:7.0:1496184940.528541:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.528544:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8450 x1568861104097840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.528551:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097840:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.528557:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.528562:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.528565:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.528567:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.528571:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.528573:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.528575:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.528579:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.528582:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.528583:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.528584:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.528585:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.528587:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.528591:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.528593:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.528595:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.528597:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.528600:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.528602:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.528604:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.528605:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.528608:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097840 00000100:00000040:7.0:1496184940.528612:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.528615:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.528627:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.528629:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8450 x1568861104097840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.528635:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097840:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.528640:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43190 00000100:00000040:7.0:1496184940.528641:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.528644:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.529259:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.529263:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.529264:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.529269:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.529277:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.529284:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.529286:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000100:00100000:7.0:1496184940.529296:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57405 00000100:00000040:7.0:1496184940.529298:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.529300:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.529307:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.529351:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.531176:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90da00 00000100:00020000:3.0:1496184940.542236:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90da00 00010000:00020000:7.0:1496184940.553295:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.553353:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.553358:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.553361:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.553366:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.553369:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.553374:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24068us (24160us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.553379:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57405 00000100:00000040:7.0:1496184940.553381:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.553385:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.553391:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.553393:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00000020:00000040:7.0:1496184940.553396:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.563821:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097872 00000020:00000040:7.0:1496184940.563824:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.563826:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.563840:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43191 00000100:00000040:7.0:1496184940.563842:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.563844:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8850 x1568861104097872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.563851:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097872:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.563857:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.563862:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.563865:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.563868:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.563871:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.563873:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.563874:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.563879:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.563882:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.563883:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.563884:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.563885:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.563887:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.563904:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.563906:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.563908:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.563910:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.563912:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.563915:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.563917:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.563918:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.563921:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097872 00000100:00000040:7.0:1496184940.563934:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.563936:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.563948:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.563951:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8850 x1568861104097872/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.563957:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097872:12345-10.37.248.124@o2ib1:8 Request procesed in 106us (183us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.563961:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43191 00000100:00000040:7.0:1496184940.563963:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.563966:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.564630:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.564633:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.564635:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.564640:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.564648:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.564655:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.564657:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000100:00100000:7.0:1496184940.564667:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57406 00000100:00000040:7.0:1496184940.564669:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.564671:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.564678:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.564721:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.566471:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90dc00 00000100:00020000:3.0:1496184940.577536:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90dc00 00010000:00020000:7.0:1496184940.588617:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.588662:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.588667:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.588670:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.588675:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.588678:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.588683:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24006us (24103us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.588688:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57406 00000100:00000040:7.0:1496184940.588690:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.588693:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.588699:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.588702:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e850. 00000020:00000040:7.0:1496184940.588705:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.598853:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097904 00000020:00000040:7.0:1496184940.598857:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.598860:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.598881:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43192 00000100:00000040:7.0:1496184940.598884:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.598888:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5050 x1568861104097904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.598923:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097904:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.598929:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.598934:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.598937:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.598939:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.598943:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.598945:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.598946:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.598951:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.598954:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.598955:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.598956:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.598957:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.598959:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.598963:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.598964:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.598966:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.598968:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.598971:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.598973:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.598975:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.598976:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.598980:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097904 00000100:00000040:7.0:1496184940.598984:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.598987:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.598999:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.599001:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5050 x1568861104097904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.599007:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097904:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (213us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.599012:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43192 00000100:00000040:7.0:1496184940.599013:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.599017:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.599585:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.599588:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.599590:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.599596:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.599603:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.599610:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.599613:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000100:00100000:7.0:1496184940.599622:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57407 00000100:00000040:7.0:1496184940.599624:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.599626:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.599633:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.599676:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.601146:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c200 00000100:00020000:3.0:1496184940.612200:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c200 00010000:00020000:7.0:1496184940.623261:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.623318:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.623323:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.623326:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.623331:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.623333:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.623339:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23706us (23802us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.623344:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57407 00000100:00000040:7.0:1496184940.623346:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.623349:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.623355:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.623358:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078c50. 00000020:00000040:7.0:1496184940.623361:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.633659:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097936 00000020:00000040:7.0:1496184940.633662:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.633664:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.633678:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43193 00000100:00000040:7.0:1496184940.633680:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.633683:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5450 x1568861104097936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.633690:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097936:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.633695:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.633701:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.633703:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.633706:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.633710:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.633712:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.633713:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.633718:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.633721:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.633722:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.633723:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.633724:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.633726:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.633730:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.633731:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.633733:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.633735:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.633738:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.633740:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.633742:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.633743:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.633747:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097936 00000100:00000040:7.0:1496184940.633751:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.633753:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.633765:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.633768:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5450 x1568861104097936/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.633773:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097936:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.633778:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43193 00000100:00000040:7.0:1496184940.633779:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.633783:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.634326:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.634329:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.634331:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.634336:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.634344:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.634351:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.634353:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000100:00100000:7.0:1496184940.634363:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57408 00000100:00000040:7.0:1496184940.634364:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.634367:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.634373:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.634418:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.635867:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184940.646911:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184940.658007:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184940.658023:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89050 x1568861104097968 msgsize 520 00000100:00100000:1.0:1496184940.658031:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184940.658033:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:00000040:7.0:1496184940.658052:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184940.658053:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104097968 00000020:02000400:7.0:1496184940.658057:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000040:3.0:1496184940.658057:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00010000:00000080:7.0:1496184940.658060:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184940.658060:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000040:7.0:1496184940.658065:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.658068:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.658074:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23701us (23793us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.658079:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57408 00000100:00000040:7.0:1496184940.658081:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.658085:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184940.658086:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6054 00000100:00000040:3.0:1496184940.658088:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184940.658091:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f050 with x1568861102817424 ext(0->1048575) 00000100:00000040:3.0:1496184940.658093:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89050 x1568861104097968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00010000:7.0:1496184940.658094:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f050. 00000020:00000040:7.0:1496184940.658098:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184940.658102:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097968:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184940.658115:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184940.658127:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184940.658131:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184940.658135:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:3.0:1496184940.658151:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184940.658154:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184940.658157:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184940.658165:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184940.658169:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.658172:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.658173:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184940.658174:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.658180:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.658185:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.658188:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184940.658191:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.658194:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.658199:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184940.658201:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184940.658204:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.658206:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184940.658214:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104097968 00000100:00000040:3.0:1496184940.658221:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.658224:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.658245:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.658247:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89050 x1568861104097968/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184940.658254:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104097968:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184940.658258:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6054 00000100:00000040:3.0:1496184940.658260:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.658266:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.659209:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.659212:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.659214:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.659219:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.659227:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.659234:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.659236:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000100:00100000:7.0:1496184940.659246:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57409 00000100:00000040:7.0:1496184940.659248:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.659250:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.659257:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.659300:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.669962:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d000 00000100:00020000:3.0:1496184940.680965:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d000 00010000:00020000:7.0:1496184940.692015:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.692072:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.692077:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.692080:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.692085:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.692088:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.692094:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32837us (32933us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.692099:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57409 00000100:00000040:7.0:1496184940.692101:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.692105:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.692110:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.692113:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077c50. 00000020:00000040:7.0:1496184940.692116:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.702965:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098000 00000020:00000040:7.0:1496184940.702968:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.702970:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.702984:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43194 00000100:00000040:7.0:1496184940.702986:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.702989:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4850 x1568861104098000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.702996:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098000:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.703002:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.703008:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.703010:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.703013:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.703017:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.703019:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.703020:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.703026:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.703029:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.703030:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.703031:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.703032:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.703034:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.703038:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.703040:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.703042:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.703044:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.703047:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.703049:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.703051:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.703052:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.703056:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098000 00000100:00000040:7.0:1496184940.703060:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.703063:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.703079:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.703082:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4850 x1568861104098000/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.703088:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098000:12345-10.37.248.124@o2ib1:8 Request procesed in 91us (171us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.703092:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43194 00000100:00000040:7.0:1496184940.703093:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.703097:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.703622:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.703625:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.703627:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.703633:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.703641:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.703648:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.703650:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000100:00100000:7.0:1496184940.703660:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57410 00000100:00000040:7.0:1496184940.703661:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.703664:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.703670:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.703714:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.705564:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c400 00000100:00020000:3.0:1496184940.705575:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c400 00010000:00020000:7.0:1496184940.705619:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.705666:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.705670:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.705673:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.705678:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.705681:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.705687:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2016us (2107us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.705692:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57410 00000100:00000040:7.0:1496184940.705694:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.705697:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.705707:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.705710:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079050. 00000020:00000040:7.0:1496184940.705713:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.733970:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098032 00000020:00000040:7.0:1496184940.733973:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.733974:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.733989:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43195 00000100:00000040:7.0:1496184940.733990:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.733993:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4c50 x1568861104098032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.734000:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098032:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.734006:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.734011:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.734014:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.734017:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.734021:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.734022:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.734024:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.734028:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.734031:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.734032:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.734033:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.734034:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.734036:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.734040:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.734042:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.734044:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.734046:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.734049:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.734051:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.734053:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.734054:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.734057:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098032 00000100:00000040:7.0:1496184940.734061:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.734064:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.734076:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.734078:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4c50 x1568861104098032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.734085:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098032:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.734089:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43195 00000100:00000040:7.0:1496184940.734090:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.734094:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.734639:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.734642:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.734644:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.734649:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.734657:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.734663:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.734666:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000100:00100000:7.0:1496184940.734675:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57411 00000100:00000040:7.0:1496184940.734677:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.734679:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.734686:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.734731:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.736467:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90e800 00000100:00020000:3.0:1496184940.736492:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90e800 00010000:00020000:7.0:1496184940.736533:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.736586:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.736591:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.736594:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.736599:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.736602:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.736607:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1921us (2010us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.736612:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57411 00000100:00000040:7.0:1496184940.736614:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.736618:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.736623:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.736626:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f450. 00000020:00000040:7.0:1496184940.736629:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.764873:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098064 00000020:00000040:7.0:1496184940.764876:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.764878:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.764906:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43196 00000100:00000040:7.0:1496184940.764908:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.764912:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4050 x1568861104098064/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.764919:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098064:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.764925:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.764931:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.764933:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.764936:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.764940:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.764941:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.764943:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.764947:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.764950:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.764951:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.764952:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.764953:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.764955:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.764959:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.764961:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.764963:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.764965:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.764968:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.764970:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.764972:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.764973:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.764976:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098064 00000100:00000040:7.0:1496184940.764980:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.764983:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.764995:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.764998:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4050 x1568861104098064/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.765004:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098064:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (183us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.765008:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43196 00000100:00000040:7.0:1496184940.765009:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.765013:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.765563:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.765566:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.765568:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.765573:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.765581:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.765587:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.765590:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000100:00100000:7.0:1496184940.765599:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57412 00000100:00000040:7.0:1496184940.765601:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.765603:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.765610:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.765654:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.767241:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ea00 00000100:00020000:3.0:1496184940.767268:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ea00 00010000:00020000:7.0:1496184940.767312:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.767366:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.767370:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.767374:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.767379:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.767382:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.767388:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1777us (1867us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.767393:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57412 00000100:00000040:7.0:1496184940.767394:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.767398:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.767403:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.767406:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078050. 00000020:00000040:7.0:1496184940.767409:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.795595:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098096 00000020:00000040:7.0:1496184940.795598:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.795600:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.795614:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43197 00000100:00000040:7.0:1496184940.795616:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.795619:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4450 x1568861104098096/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.795626:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098096:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.795632:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.795637:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.795640:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.795642:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.795646:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.795648:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.795650:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.795654:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.795657:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.795658:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.795659:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.795660:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.795662:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.795666:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.795667:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.795670:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.795672:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.795674:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.795676:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.795678:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.795679:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.795683:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098096 00000100:00000040:7.0:1496184940.795687:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.795689:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.795701:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.795704:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4450 x1568861104098096/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.795710:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098096:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.795714:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43197 00000100:00000040:7.0:1496184940.795716:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.795719:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.796280:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.796284:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.796285:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.796290:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.796298:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.796305:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.796307:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000100:00100000:7.0:1496184940.796317:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57413 00000100:00000040:7.0:1496184940.796319:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.796321:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.796328:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.796372:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.797831:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90ec00 00000100:00020000:3.0:1496184940.809004:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90ec00 00010000:00020000:7.0:1496184940.820060:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.820117:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.820122:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.820125:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.820130:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.820133:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.820139:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23811us (23903us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.820144:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57413 00000100:00000040:7.0:1496184940.820145:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.820149:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.820155:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.820158:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079850. 00000020:00000040:7.0:1496184940.820161:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.830574:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098128 00000020:00000040:7.0:1496184940.830577:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.830579:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.830593:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43198 00000100:00000040:7.0:1496184940.830595:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.830597:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3050 x1568861104098128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.830605:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098128:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.830610:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.830615:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.830618:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.830621:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.830625:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.830626:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.830628:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.830632:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.830635:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.830636:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.830637:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.830638:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.830640:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.830644:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.830646:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.830648:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.830650:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.830652:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.830655:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.830657:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.830658:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.830661:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098128 00000100:00000040:7.0:1496184940.830665:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.830668:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.830680:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.830682:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3050 x1568861104098128/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.830688:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098128:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (168us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.830693:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43198 00000100:00000040:7.0:1496184940.830694:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.830697:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.831265:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.831268:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.831270:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.831275:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.831283:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.831290:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.831292:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000100:00100000:7.0:1496184940.831302:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57414 00000100:00000040:7.0:1496184940.831304:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.831306:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.831313:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.831355:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.833142:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184940.844221:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184940.855334:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184940.855356:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286050 x1568861104098160 msgsize 520 00000100:00100000:1.0:1496184940.855365:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184940.855366:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184940.855386:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098160 00000020:00000040:3.0:1496184940.855390:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.855393:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.855394:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:02000400:7.0:1496184940.855398:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.855401:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.855406:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.855408:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.855414:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24102us (24201us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184940.855418:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6055 00000100:00100000:7.0:1496184940.855420:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57414 00000100:00000040:3.0:1496184940.855421:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:7.0:1496184940.855422:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00002000:00100000:7.0:1496184940.855426:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184940.855426:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286050 x1568861104098160/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.855432:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.855434:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077050. 00000100:00100000:3.0:1496184940.855435:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104098160:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.855438:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.855447:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184940.855459:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184940.855464:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184940.855467:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:3.0:1496184940.855476:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184940.855486:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184940.855488:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184940.855497:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184940.855501:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.855503:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.855504:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184940.855505:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184940.855508:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.855514:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.855517:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184940.855520:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.855525:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184940.855530:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184940.855533:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184940.855536:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184940.855537:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184940.855546:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098160 00000100:00000040:3.0:1496184940.855554:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.855557:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.855577:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184940.855580:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286050 x1568861104098160/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184940.855586:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098160:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184940.855592:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6055 00000100:00000040:3.0:1496184940.855594:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184940.855600:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.856524:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.856527:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.856529:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.856536:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.856544:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.856550:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.856553:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000100:00100000:7.0:1496184940.856563:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57415 00000100:00000040:7.0:1496184940.856565:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.856568:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.856574:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.856618:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.857956:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90de00 00000100:00020000:3.0:1496184940.868981:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90de00 00010000:00020000:7.0:1496184940.880064:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.880110:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.880114:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.880118:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.880123:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.880125:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.880131:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23557us (23652us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.880137:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57415 00000100:00000040:7.0:1496184940.880138:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.880142:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.880152:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.880155:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078450. 00000020:00000040:7.0:1496184940.880158:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.891832:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098192 00000020:00000040:7.0:1496184940.891835:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.891837:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.891851:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43199 00000100:00000040:7.0:1496184940.891853:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.891855:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3c50 x1568861104098192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.891862:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098192:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.891869:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.891874:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.891877:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.891879:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.891884:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.891885:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.891887:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.891904:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.891908:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.891909:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.891910:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.891911:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.891913:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.891918:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.891920:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.891922:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.891924:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.891927:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.891929:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.891931:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.891932:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.891936:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098192 00000100:00000040:7.0:1496184940.891940:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.891943:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.891955:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.891958:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3c50 x1568861104098192/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.891964:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098192:12345-10.37.248.124@o2ib1:8 Request procesed in 101us (179us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.891968:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43199 00000100:00000040:7.0:1496184940.891970:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.891973:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.892526:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.892530:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.892531:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.892537:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.892545:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.892551:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.892554:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000100:00100000:7.0:1496184940.892563:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57416 00000100:00000040:7.0:1496184940.892565:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.892568:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.892574:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.892619:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.894288:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fc00 00000100:00020000:3.0:1496184940.905333:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fc00 00010000:00020000:7.0:1496184940.916394:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.916440:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.916445:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.916448:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.916453:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.916455:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.916461:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23887us (23983us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.916466:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57416 00000100:00000040:7.0:1496184940.916468:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.916472:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.916477:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.916480:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077850. 00000020:00000040:7.0:1496184940.916483:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.927184:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098224 00000020:00000040:7.0:1496184940.927187:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.927189:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.927203:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43200 00000100:00000040:7.0:1496184940.927205:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.927208:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3850 x1568861104098224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.927215:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098224:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.927221:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.927226:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.927229:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.927231:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.927235:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.927237:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.927238:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.927243:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.927246:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.927247:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.927248:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.927249:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.927251:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.927255:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.927256:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.927259:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.927261:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.927263:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.927265:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.927267:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.927268:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.927272:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098224 00000100:00000040:7.0:1496184940.927276:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.927279:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.927291:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.927293:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3850 x1568861104098224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.927299:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098224:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (164us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.927304:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43200 00000100:00000040:7.0:1496184940.927305:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.927308:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.927886:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.927902:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.927904:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.927909:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.927917:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.927924:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.927926:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000100:00100000:7.0:1496184940.927935:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57417 00000100:00000040:7.0:1496184940.927937:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.927939:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.927946:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.927989:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.929560:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90c000 00000100:00020000:3.0:1496184940.940634:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90c000 00000100:00000040:2.0:1496184940.941779:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee6c50 x1568861152397616 msgsize 224 00000100:00100000:2.0:1496184940.941788:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184940.941789:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184940.941837:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397616 00000020:00000040:1.0:1496184940.941842:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:1.0:1496184940.941845:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:1.0:1496184940.941856:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184940.941875:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 956 00000100:00000040:1.0:1496184940.941877:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:1.0:1496184940.941882:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee6c50 x1568861152397616/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:691/0 lens 224/0 e 0 to 0 dl 1496184946 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184940.941902:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397616:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184940.941926:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:1.0:1496184940.941933:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397616 00000100:00000040:1.0:1496184940.941941:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184940.941946:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184940.941968:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184940.941971:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee6c50 x1568861152397616/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:691/0 lens 224/336 e 0 to 0 dl 1496184946 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184940.941977:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397616:12345-10.37.248.197@o2ib1:13 Request procesed in 76us (197us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184940.941982:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 956 00000100:00000040:1.0:1496184940.941984:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:1.0:1496184940.941991:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00010000:00020000:7.0:1496184940.951687:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.951746:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.951751:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.951754:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.951759:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.951762:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.951768:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23822us (23927us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.951773:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57417 00000100:00000040:7.0:1496184940.951775:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.951778:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.951784:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.951787:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076050. 00000020:00000040:7.0:1496184940.951790:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.962368:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098256 00000020:00000040:7.0:1496184940.962372:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.962373:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.962388:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43201 00000100:00000040:7.0:1496184940.962389:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.962392:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3450 x1568861104098256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.962399:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098256:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.962405:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.962411:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.962413:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.962416:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.962420:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.962422:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.962423:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.962427:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.962430:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.962432:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.962432:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.962433:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.962435:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.962439:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.962441:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.962443:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.962445:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.962448:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.962450:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.962452:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.962453:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.962457:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098256 00000100:00000040:7.0:1496184940.962461:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.962464:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.962476:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.962478:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3450 x1568861104098256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.962484:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098256:12345-10.37.248.124@o2ib1:8 Request procesed in 85us (160us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.962488:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43201 00000100:00000040:7.0:1496184940.962490:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.962493:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.963100:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.963103:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.963105:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.963111:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.963119:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.963125:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.963127:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000100:00100000:7.0:1496184940.963137:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57418 00000100:00000040:7.0:1496184940.963139:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.963141:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.963147:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.963191:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184940.964744:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90d600 00000100:00020000:3.0:1496184940.975781:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90d600 00010000:00020000:7.0:1496184940.986834:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184940.986904:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184940.986909:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184940.986912:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184940.986917:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184940.986919:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184940.986925:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23778us (23870us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184940.986930:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57418 00000100:00000040:7.0:1496184940.986932:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184940.986936:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184940.986944:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076c50 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.986947:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076c50. 00000020:00000040:7.0:1496184940.986950:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.997773:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098288 00000020:00000040:7.0:1496184940.997776:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.997778:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184940.997792:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43202 00000100:00000040:7.0:1496184940.997794:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.997796:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2450 x1568861104098288/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184960 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.997804:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098288:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184940.997809:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184940.997815:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184940.997817:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184940.997820:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184940 last 1496184939 00000001:00000040:7.0:1496184940.997824:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184940.997825:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184940.997827:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184940.997831:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184940.997834:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.997835:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.997836:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184940.997837:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.997839:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.997843:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.997845:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184940.997847:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.997849:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184940.997852:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184940.997854:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184940.997856:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184940.997857:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184940.997860:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098288 00000100:00000040:7.0:1496184940.997864:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.997867:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.997879:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184940.997881:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2450 x1568861104098288/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184960 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184940.997887:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098288:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (161us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184940.997903:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43202 00000100:00000040:7.0:1496184940.997905:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184940.997908:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184940.998533:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184940.998536:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184940.998538:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184940.998543:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184940.998551:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184940.998558:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184940.998560:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000100:00100000:7.0:1496184940.998570:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57419 00000100:00000040:7.0:1496184940.998572:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184940.998574:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/0 e 0 to 0 dl 1496184983 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184940.998580:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184940.998624:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184941.000715:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90f000 00000100:00020000:3.0:1496184941.011753:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90f000 00010000:00020000:7.0:1496184941.022803:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184941.022861:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184941.022865:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184941.022868:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184941.022874:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184941.022876:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184941.022882:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24301us (24394us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184941.022887:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57419 00000100:00000040:7.0:1496184941.022889:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184941.022905:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:728/0 lens 608/448 e 0 to 0 dl 1496184983 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184941.022911:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075050 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.022913:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075050. 00000020:00000040:7.0:1496184941.022917:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184941.033242:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098320 00000020:00000040:7.0:1496184941.033247:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184941.033248:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:7.0:1496184941.033263:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43203 00000100:00000040:7.0:1496184941.033265:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184941.033268:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2c50 x1568861104098320/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:7.0:1496184941.033275:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098320:12345-10.37.248.124@o2ib1:8 00000020:00000040:7.0:1496184941.033281:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:7.0:1496184941.033286:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:7.0:1496184941.033289:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:7.0:1496184941.033291:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:7.0:1496184941.033295:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:7.0:1496184941.033297:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:7.0:1496184941.033298:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:7.0:1496184941.033303:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:7.0:1496184941.033306:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184941.033307:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184941.033308:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:7.0:1496184941.033309:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184941.033311:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184941.033315:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184941.033316:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:7.0:1496184941.033318:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184941.033320:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:7.0:1496184941.033323:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:7.0:1496184941.033325:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184941.033327:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:7.0:1496184941.033328:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:7.0:1496184941.033332:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098320 00000100:00000040:7.0:1496184941.033336:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184941.033338:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:7.0:1496184941.033350:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:7.0:1496184941.033353:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2c50 x1568861104098320/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:7.0:1496184941.033359:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098320:12345-10.37.248.124@o2ib1:8 Request procesed in 84us (162us total) trans 0 rc 0/0 00000100:00100000:7.0:1496184941.033363:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43203 00000100:00000040:7.0:1496184941.033365:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:7.0:1496184941.033368:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184941.033911:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184941.033915:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184941.033916:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184941.033921:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184941.033929:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184941.033936:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.033938:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000100:00100000:7.0:1496184941.033948:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57420 00000100:00000040:7.0:1496184941.033950:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184941.033952:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184941.033959:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184941.034002:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184941.035726:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184941.046774:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184941.057864:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000040:1.0:1496184941.057924:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285450 x1568861104098352 msgsize 520 00000100:00100000:1.0:1496184941.057933:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000020:00000040:7.0:1496184941.057934:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000040:1.0:1496184941.057934:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000020:02000400:7.0:1496184941.057939:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184941.057942:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184941.057948:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184941.057950:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184941.057956:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23997us (24090us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184941.057961:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57420 00000100:00100000:3.0:1496184941.057961:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098352 00000100:00000040:7.0:1496184941.057963:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184941.057966:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184941.057967:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184941.057971:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00002000:00100000:7.0:1496184941.057972:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.057975:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075450. 00000020:00000040:7.0:1496184941.057978:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184941.057998:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6056 00000100:00000040:3.0:1496184941.058001:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.058006:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285450 x1568861104098352/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184941.058015:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098352:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184941.058027:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184941.058047:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184941.058052:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184941.058055:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:3.0:1496184941.058064:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184941.058067:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184941.058070:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184941.058078:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184941.058083:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.058085:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.058086:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184941.058087:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.058090:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.058096:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.058098:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184941.058101:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.058103:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.058109:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184941.058112:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184941.058115:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.058117:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184941.058125:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098352 00000100:00000040:3.0:1496184941.058133:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.058137:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.058157:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.058160:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285450 x1568861104098352/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184941.058166:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098352:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (241us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184941.058171:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6056 00000100:00000040:3.0:1496184941.058173:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.058178:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184941.059054:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184941.059059:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184941.059062:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184941.059070:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184941.059082:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184941.059092:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.059095:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000100:00100000:7.0:1496184941.059117:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57421 00000100:00000040:7.0:1496184941.059119:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184941.059121:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184941.059132:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184941.059176:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184941.070026:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184941.081043:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184941.092086:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184941.092143:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184941.092148:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184941.092151:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184941.092156:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184941.092158:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184941.092164:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33032us (33171us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184941.092169:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57421 00000100:00000040:7.0:1496184941.092171:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184941.092175:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184941.092180:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074450 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.092183:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074450. 00000020:00000040:7.0:1496184941.092186:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:1.0:1496184941.102507:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286450 x1568861104098384 msgsize 520 00000100:00100000:1.0:1496184941.102515:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184941.102517:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184941.102559:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098384 00000020:00000040:3.0:1496184941.102562:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184941.102565:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184941.102588:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6057 00000100:00000040:3.0:1496184941.102590:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.102594:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286450 x1568861104098384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184941.102602:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098384:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184941.102612:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184941.102621:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184941.102625:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184941.102628:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:3.0:1496184941.102635:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184941.102640:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184941.102641:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184941.102647:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184941.102651:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.102652:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.102653:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184941.102654:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.102657:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.102661:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.102663:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184941.102666:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.102668:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.102672:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184941.102675:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184941.102677:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.102679:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184941.102686:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098384 00000100:00000040:3.0:1496184941.102692:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.102695:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.102712:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.102715:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286450 x1568861104098384/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184941.102721:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098384:12345-10.37.248.124@o2ib1:8 Request procesed in 119us (213us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184941.102726:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6057 00000100:00000040:3.0:1496184941.102727:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.102731:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00100000:7.0:1496184941.103280:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000040:7.0:1496184941.103283:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:7.0:1496184941.103285:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00000040:7.0:1496184941.103292:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00002000:00100000:7.0:1496184941.103300:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00100000:7.0:1496184941.103311:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.103314:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000100:00100000:7.0:1496184941.103324:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57422 00000100:00000040:7.0:1496184941.103325:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:7.0:1496184941.103328:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/0 e 0 to 0 dl 1496184984 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00100000:7.0:1496184941.103334:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000020:00000040:7.0:1496184941.103375:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00020000:3.0:1496184941.105538:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff88100f90fa00 00000100:00020000:3.0:1496184941.105564:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff88100f90fa00 00010000:00020000:7.0:1496184941.105607:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000040:7.0:1496184941.105661:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:02000400:7.0:1496184941.105665:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00010000:00000080:7.0:1496184941.105668:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:7.0:1496184941.105674:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:7.0:1496184941.105676:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:7.0:1496184941.105682:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2348us (2449us total) trans 0 rc -110/-999 00000100:00100000:7.0:1496184941.105687:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57422 00000100:00000040:7.0:1496184941.105689:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00002000:00100000:7.0:1496184941.105692:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:729/0 lens 608/448 e 0 to 0 dl 1496184984 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00100000:7.0:1496184941.105698:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00002000:00010000:7.0:1496184941.105701:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000020:00000040:7.0:1496184941.105704:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184941.134295:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2850 x1568861104098416 msgsize 520 00000100:00100000:7.0:1496184941.134302:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.134303:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.134978:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.134985:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.134986:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.136701:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.136711:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:0.0:1496184941.167748:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:0.0:1496184941.167764:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.195263:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2050 x1568861104098480 msgsize 520 00000100:00100000:7.0:1496184941.195270:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.195272:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.196015:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.196030:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.196032:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.197688:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.208814:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.230147:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1850 x1568861104098512 msgsize 520 00000100:00100000:7.0:1496184941.230154:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.230155:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.232515:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.243571:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:0.0:1496184941.254644:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285850 x1568861104098544 msgsize 520 00000100:00100000:0.0:1496184941.254652:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184941.254654:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184941.254675:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098544 00000020:00000040:3.0:1496184941.254687:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184941.254691:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184941.254716:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6058 00000100:00000040:3.0:1496184941.254718:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184941.254724:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285850 x1568861104098544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184941.254732:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104098544:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184941.254744:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184941.254756:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184941.254761:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184941.254765:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:3.0:1496184941.254774:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184941.254777:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184941.254779:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184941.254788:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184941.254792:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.254795:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.254796:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184941.254797:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.254803:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.254808:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.254811:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184941.254814:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.254816:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.254821:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184941.254824:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184941.254826:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.254828:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184941.254844:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098544 00000100:00000040:3.0:1496184941.254851:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.254855:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.254875:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.254877:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285850 x1568861104098544/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184941.254884:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098544:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184941.254898:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6058 00000100:00000040:3.0:1496184941.254900:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.254906:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184941.255822:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.255829:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.255831:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.257452:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.268464:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184941.293180:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0450 x1568861104098576 msgsize 520 00000100:00100000:7.0:1496184941.293187:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.293188:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.293834:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.293841:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.293842:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.295751:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.306780:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.328860:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1450 x1568861104098608 msgsize 520 00000100:00100000:7.0:1496184941.328867:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.328875:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.331548:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.342603:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.365569:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.365576:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.365578:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.367260:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.378307:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184941.399913:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1050 x1568861104098672 msgsize 520 00000100:00100000:7.0:1496184941.399920:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.399921:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.402473:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.413511:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184941.434733:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0c50 x1568861104098704 msgsize 520 00000100:00100000:7.0:1496184941.434740:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.434749:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.437126:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.448167:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:1.0:1496184941.459263:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc285c50 x1568861104098736 msgsize 520 00000100:00100000:1.0:1496184941.459272:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184941.459273:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184941.459293:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098736 00000020:00000040:3.0:1496184941.459297:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184941.459300:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184941.459325:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6059 00000100:00000040:3.0:1496184941.459328:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.459333:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc285c50 x1568861104098736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184941.459342:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104098736:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184941.459354:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184941.459367:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184941.459371:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184941.459375:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:3.0:1496184941.459383:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184941.459386:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184941.459389:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184941.459404:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184941.459408:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.459411:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.459412:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184941.459413:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184941.459416:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.459422:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.459425:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184941.459428:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.459433:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184941.459438:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184941.459441:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184941.459444:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184941.459445:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184941.459454:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098736 00000100:00000040:3.0:1496184941.459461:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.459465:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.459485:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184941.459488:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc285c50 x1568861104098736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184941.459494:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098736:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184941.459499:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6059 00000100:00000040:3.0:1496184941.459501:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184941.459507:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184941.470331:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.481327:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:3.0:1496184941.502727:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287850 x1568861104098768 msgsize 520 00000100:00100000:3.0:1496184941.502734:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184941.502736:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184941.502799:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098768 00000020:00000040:0.0:1496184941.502805:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184941.502808:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184941.502846:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6060 00000100:00000040:0.0:1496184941.502850:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184941.502854:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287850 x1568861104098768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184941.502862:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098768:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184941.502871:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184941.502879:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184941.502882:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184941.502885:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:0.0:1496184941.502902:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184941.502904:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184941.502906:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184941.502912:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184941.502916:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184941.502917:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184941.502918:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184941.502920:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184941.502922:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184941.502927:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184941.502928:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184941.502931:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184941.502933:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184941.502937:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184941.502940:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184941.502942:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184941.502943:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184941.502948:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098768 00000100:00000040:0.0:1496184941.502955:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184941.502958:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184941.502974:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184941.502976:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287850 x1568861104098768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184941.502983:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098768:12345-10.37.248.124@o2ib1:8 Request procesed in 121us (255us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184941.502987:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6060 00000100:00000040:0.0:1496184941.502989:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184941.502993:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184941.505261:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.505272:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.533680:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd8c50 x1568861104098800 msgsize 520 00000100:00100000:7.0:1496184941.533687:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.533688:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.534355:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.534362:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.534364:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.535987:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.535998:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184941.564234:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104098832 msgsize 520 00000100:00100000:7.0:1496184941.564241:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.564242:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.564946:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.564953:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.564954:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.566590:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.566600:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184941.594616:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104098864 msgsize 520 00000100:00100000:7.0:1496184941.594623:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.594624:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.595378:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.595385:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.595394:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.596874:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.608023:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184941.629655:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9050 x1568861104098896 msgsize 520 00000100:00100000:7.0:1496184941.629662:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.629663:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.630363:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.630371:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.630372:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.632166:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.643245:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:1.0:1496184941.654380:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286c50 x1568861104098928 msgsize 520 00000100:00100000:1.0:1496184941.654389:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184941.654391:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184941.654446:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104098928 00000020:00000040:2.0:1496184941.654451:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184941.654454:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184941.654481:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6061 00000100:00000040:2.0:1496184941.654483:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.654488:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286c50 x1568861104098928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184941.654497:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098928:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184941.654516:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184941.654528:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184941.654533:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184941.654537:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:2.0:1496184941.654546:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184941.654549:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184941.654551:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184941.654560:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184941.654567:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.654569:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.654570:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184941.654571:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.654574:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.654580:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.654583:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184941.654586:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.654588:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.654593:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184941.654596:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184941.654599:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.654600:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184941.654609:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104098928 00000100:00000040:2.0:1496184941.654617:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.654620:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.654640:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.654643:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286c50 x1568861104098928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184941.654649:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104098928:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (268us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184941.654654:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6061 00000100:00000040:2.0:1496184941.654656:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.654662:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184941.655559:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.655566:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.655568:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.657219:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.668222:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184941.693197:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9c50 x1568861104098960 msgsize 520 00000100:00100000:7.0:1496184941.693204:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.693205:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.693859:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.693866:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.693867:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.695454:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.706495:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.730676:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.741738:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.765582:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.776617:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.800972:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.811990:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184941.833566:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdac50 x1568861104099088 msgsize 520 00000100:00100000:7.0:1496184941.833573:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.833574:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.834267:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.834274:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.834279:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.835928:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.846978:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184941.858073:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287450 x1568861104099120 msgsize 520 00000100:00100000:1.0:1496184941.858082:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184941.858084:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184941.858134:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099120 00000020:00000040:2.0:1496184941.858139:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184941.858142:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184941.858169:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6062 00000100:00000040:2.0:1496184941.858172:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.858179:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287450 x1568861104099120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184941.858188:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099120:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184941.858207:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184941.858219:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184941.858224:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184941.858228:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:2.0:1496184941.858236:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184941.858240:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184941.858242:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184941.858251:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184941.858255:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.858257:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.858258:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184941.858259:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.858262:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.858268:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.858271:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184941.858274:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.858276:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.858281:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184941.858284:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184941.858286:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.858288:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184941.858297:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099120 00000100:00000040:2.0:1496184941.858304:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.858309:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.858330:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.858332:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287450 x1568861104099120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184941.858339:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099120:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (265us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184941.858344:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6062 00000100:00000040:2.0:1496184941.858346:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.858351:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184941.859183:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.859190:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.859192:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.870172:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.881177:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:3.0:1496184941.903092:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee5050 x1568861104099152 msgsize 520 00000100:00100000:3.0:1496184941.903100:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184941.903101:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184941.903117:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099152 00000020:00000040:2.0:1496184941.903121:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184941.903124:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184941.903143:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6063 00000100:00000040:2.0:1496184941.903146:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.903149:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee5050 x1568861104099152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184961 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184941.903157:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099152:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184941.903166:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184941.903175:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184941.903178:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184941.903181:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184941 last 1496184941 00000001:00000040:2.0:1496184941.903187:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184941.903189:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184941.903190:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184941.903196:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184941.903200:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.903201:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.903204:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184941.903205:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184941.903208:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.903213:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.903215:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184941.903218:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.903220:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184941.903224:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184941.903227:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184941.903229:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184941.903230:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184941.903236:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099152 00000100:00000040:2.0:1496184941.903242:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.903245:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.903266:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184941.903269:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee5050 x1568861104099152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184961 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184941.903275:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099152:12345-10.37.248.124@o2ib1:8 Request procesed in 119us (182us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184941.903280:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6063 00000100:00000040:2.0:1496184941.903282:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184941.903286:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184941.903814:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.903821:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.903822:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.905659:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.905670:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184941.936692:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184941.936718:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184941.965183:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104099216 msgsize 520 00000100:00100000:7.0:1496184941.965190:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.965191:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184941.965805:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184941.965812:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184941.965813:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184941.967404:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.967429:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184941.997931:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.009068:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:0.0:1496184942.030543:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc286850 x1568861104099280 msgsize 520 00000100:00100000:0.0:1496184942.030552:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184942.030553:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184942.030572:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099280 00000020:00000040:2.0:1496184942.030575:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184942.030585:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184942.030607:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6064 00000100:00000040:2.0:1496184942.030610:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184942.030614:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc286850 x1568861104099280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184942.030628:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099280:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184942.030639:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184942.030649:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184942.030653:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184942.030656:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:2.0:1496184942.030663:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184942.030666:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184942.030667:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184942.030675:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184942.030679:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184942.030681:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184942.030682:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184942.030683:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184942.030686:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184942.030691:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184942.030694:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184942.030697:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184942.030699:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184942.030704:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184942.030707:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184942.030709:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184942.030710:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184942.030718:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099280 00000100:00000040:2.0:1496184942.030725:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184942.030728:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184942.030748:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184942.030751:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc286850 x1568861104099280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184942.030757:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099280:12345-10.37.248.124@o2ib1:8 Request procesed in 129us (212us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184942.030762:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6064 00000100:00000040:2.0:1496184942.030763:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184942.030768:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184942.031259:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.031266:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.031268:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.033044:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.044121:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:1.0:1496184942.055217:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807fc287050 x1568861104099312 msgsize 520 00000100:00100000:1.0:1496184942.055225:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184942.055234:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184942.055245:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099312 00000020:00000040:3.0:1496184942.055247:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184942.055252:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184942.055267:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6065 00000100:00000040:3.0:1496184942.055269:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184942.055272:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807fc287050 x1568861104099312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184942.055279:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104099312:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184942.055287:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184942.055293:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184942.055297:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184942.055299:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:3.0:1496184942.055304:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184942.055306:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184942.055307:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184942.055312:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184942.055316:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.055317:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000040:3.0:1496184942.055318:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000040:3.0:1496184942.055319:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.055322:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.055326:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.055328:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184942.055330:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.055332:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.055335:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184942.055338:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184942.055340:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.055343:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184942.055347:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099312 00000100:00000040:3.0:1496184942.055352:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.055355:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.055368:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.055371:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807fc287050 x1568861104099312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184942.055383:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099312:12345-10.37.248.124@o2ib1:8 Request procesed in 104us (165us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184942.055388:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6065 00000100:00000040:3.0:1496184942.055390:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.055394:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184942.058115:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.069105:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.095902:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.106915:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.129405:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.129412:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.129413:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.131143:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.142202:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.163662:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc450 x1568861104099408 msgsize 520 00000100:00100000:7.0:1496184942.163669:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.163670:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.166205:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.177243:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.199306:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc850 x1568861104099440 msgsize 520 00000100:00100000:7.0:1496184942.199313:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.199314:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.200032:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.200039:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.200040:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.201711:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.212745:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.236693:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.247723:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:1.0:1496184942.258758:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4c50 x1568861104099504 msgsize 520 00000100:00100000:1.0:1496184942.258767:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184942.258769:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184942.258790:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099504 00000020:00000040:3.0:1496184942.258802:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184942.258806:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184942.258831:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6066 00000100:00000040:3.0:1496184942.258833:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184942.258838:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4c50 x1568861104099504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184942.258849:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104099504:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184942.258861:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184942.258873:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184942.258878:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184942.258882:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:3.0:1496184942.258897:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184942.258900:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184942.258902:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184942.258910:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184942.258915:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.258917:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.258918:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184942.258919:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.258922:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.258927:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.258930:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184942.258933:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.258935:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.258941:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184942.258944:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184942.258953:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.258954:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184942.258963:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099504 00000100:00000040:3.0:1496184942.258970:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.258974:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.258995:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.258998:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4c50 x1568861104099504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184942.259004:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099504:12345-10.37.248.124@o2ib1:8 Request procesed in 155us (246us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184942.259010:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6066 00000100:00000040:3.0:1496184942.259012:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.259017:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184942.259987:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.259994:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.259996:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.270118:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.281119:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.302630:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd050 x1568861104099536 msgsize 520 00000100:00100000:7.0:1496184942.302637:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.302638:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.303268:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.303275:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.303276:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.305207:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.305218:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.333275:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd450 x1568861104099568 msgsize 520 00000100:00100000:7.0:1496184942.333282:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.333283:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.333975:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.333982:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.333983:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.335670:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.335695:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.366168:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.366191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:2.0:1496184942.381776:0:6132:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee7050 x1568861152397632 msgsize 224 00000100:00100000:2.0:1496184942.381784:0:6132:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184942.381785:0:6132:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184942.381831:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397632 00000020:00000040:1.0:1496184942.381835:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000040:1.0:1496184942.381838:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000100:00000040:1.0:1496184942.381847:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184942.381862:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 957 00000100:00000040:1.0:1496184942.381865:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000040:1.0:1496184942.381869:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee7050 x1568861152397632/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:693/0 lens 224/0 e 0 to 0 dl 1496184948 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184942.381877:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397632:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184942.381903:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 9758656 pending 0 free 80206888960 avail 76094111744 00010000:00000040:1.0:1496184942.381909:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397632 00000100:00000040:1.0:1496184942.381916:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184942.381921:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184942.381939:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184942.381942:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee7050 x1568861152397632/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:693/0 lens 224/336 e 0 to 0 dl 1496184948 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184942.381948:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397632:12345-10.37.248.197@o2ib1:13 Request procesed in 72us (172us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184942.381953:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 957 00000100:00000040:1.0:1496184942.381955:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000020:00000040:1.0:1496184942.381960:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 00000100:00020000:3.0:1496184942.396102:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.407222:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.428519:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde450 x1568861104099664 msgsize 520 00000100:00100000:7.0:1496184942.428526:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.428528:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.429206:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.429213:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.429221:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.431044:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.442122:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:2.0:1496184942.453232:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4850 x1568861104099696 msgsize 520 00000100:00100000:2.0:1496184942.453241:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184942.453243:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184942.453263:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099696 00000020:00000040:3.0:1496184942.453266:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184942.453270:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184942.453294:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6067 00000100:00000040:3.0:1496184942.453296:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184942.453302:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4850 x1568861104099696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184942.453310:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104099696:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184942.453322:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184942.453338:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184942.453343:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184942.453346:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:3.0:1496184942.453354:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184942.453365:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184942.453368:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184942.453376:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184942.453381:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.453383:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.453384:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184942.453385:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.453388:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.453394:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.453397:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184942.453400:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.453402:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.453407:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184942.453410:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184942.453413:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.453414:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184942.453423:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099696 00000100:00000040:3.0:1496184942.453430:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.453434:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.453454:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.453457:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4850 x1568861104099696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184942.453463:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099696:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184942.453469:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6067 00000100:00000040:3.0:1496184942.453471:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.453477:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184942.454433:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.454440:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.454441:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.456052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.467040:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.492834:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde050 x1568861104099728 msgsize 520 00000100:00100000:7.0:1496184942.492841:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.492843:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.495291:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.506316:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.530826:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.541878:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.563380:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdec50 x1568861104099792 msgsize 520 00000100:00100000:7.0:1496184942.563387:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.563388:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.564042:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.564049:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.564050:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.566034:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.577074:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.598574:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf050 x1568861104099824 msgsize 520 00000100:00100000:7.0:1496184942.598581:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.598582:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.601015:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.612039:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.633620:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf450 x1568861104099856 msgsize 520 00000100:00100000:7.0:1496184942.633627:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.633628:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.636213:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.647253:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:1.0:1496184942.658345:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3c50 x1568861104099888 msgsize 520 00000100:00100000:1.0:1496184942.658354:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184942.658358:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184942.658379:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104099888 00000020:00000040:3.0:1496184942.658382:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184942.658386:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184942.658411:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6068 00000100:00000040:3.0:1496184942.658413:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.658419:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3c50 x1568861104099888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184942.658428:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099888:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184942.658440:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184942.658452:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184942.658457:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184942.658461:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:3.0:1496184942.658476:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184942.658482:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184942.658485:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184942.658493:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184942.658497:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.658499:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.658501:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184942.658502:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.658505:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.658510:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.658513:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184942.658516:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.658519:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.658524:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184942.658527:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184942.658529:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.658531:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184942.658539:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104099888 00000100:00000040:3.0:1496184942.658547:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.658551:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.658570:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.658572:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3c50 x1568861104099888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184942.658578:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104099888:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184942.658584:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6068 00000100:00000040:3.0:1496184942.658586:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.658591:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184942.670002:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.681015:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.702595:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf850 x1568861104099920 msgsize 520 00000100:00100000:7.0:1496184942.702602:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.702603:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.703295:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.703303:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.703304:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.705029:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.705055:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.733493:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c9050 x1568861104099952 msgsize 520 00000100:00100000:7.0:1496184942.733500:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.733501:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.734162:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.734169:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.734171:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.735923:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.735934:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.766740:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.766766:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.794958:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8050 x1568861104100016 msgsize 520 00000100:00100000:7.0:1496184942.794965:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.794966:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184942.795658:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.795665:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.795669:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.797206:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.808339:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.832050:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.843119:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:1.0:1496184942.854253:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4450 x1568861104100080 msgsize 520 00000100:00100000:1.0:1496184942.854262:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184942.854264:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184942.854283:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100080 00000020:00000040:3.0:1496184942.854287:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184942.854290:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184942.854315:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6069 00000100:00000040:3.0:1496184942.854317:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.854323:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4450 x1568861104100080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184962 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184942.854331:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104100080:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184942.854344:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184942.854357:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184942.854361:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184942.854365:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184942 last 1496184941 00000001:00000040:3.0:1496184942.854373:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184942.854376:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184942.854379:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184942.854394:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184942.854401:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.854403:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.854404:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184942.854405:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184942.854409:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.854414:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.854417:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184942.854420:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.854422:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184942.854427:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184942.854430:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184942.854432:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184942.854434:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184942.854442:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100080 00000100:00000040:3.0:1496184942.854450:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.854453:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.854473:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184942.854476:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4450 x1568861104100080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184962 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184942.854482:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100080:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (228us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184942.854487:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6069 00000100:00000040:3.0:1496184942.854489:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184942.854494:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184942.855387:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.855394:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.855395:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.857026:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.868033:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184942.893406:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7c50 x1568861104100112 msgsize 520 00000100:00100000:7.0:1496184942.893413:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.893415:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.895777:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.906799:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.931208:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184942.942268:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184942.965087:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184942.965094:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184942.965102:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184942.966856:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184942.977899:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.002146:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.013169:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.035286:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.035293:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.035294:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.037101:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184943.048119:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:1.0:1496184943.059203:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3450 x1568861104100272 msgsize 520 00000100:00100000:1.0:1496184943.059211:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184943.059213:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.059233:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100272 00000020:00000040:3.0:1496184943.059237:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.059240:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184943.059266:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6070 00000100:00000040:3.0:1496184943.059268:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184943.059274:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3450 x1568861104100272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.059283:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104100272:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.059295:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.059308:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.059313:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.059316:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.059325:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.059328:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.059330:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.059346:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.059350:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.059352:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.059353:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.059355:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.059361:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.059367:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.059369:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.059372:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.059374:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.059380:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.059383:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.059385:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.059387:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.059395:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100272 00000100:00000040:3.0:1496184943.059402:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.059406:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.059427:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.059429:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3450 x1568861104100272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.059436:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100272:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (232us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.059441:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6070 00000100:00000040:3.0:1496184943.059443:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.059448:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184943.060366:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.060373:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.060374:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.070087:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184943.081094:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:1.0:1496184943.102887:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3850 x1568861104100304 msgsize 520 00000100:00100000:1.0:1496184943.102903:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184943.102905:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.102949:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100304 00000020:00000040:3.0:1496184943.102953:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.102956:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184943.102978:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6071 00000100:00000040:3.0:1496184943.102980:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.102984:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3850 x1568861104100304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.102999:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100304:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.103009:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.103018:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.103022:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.103025:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.103031:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.103034:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.103035:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.103042:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.103045:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.103047:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.103048:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.103049:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.103051:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.103056:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.103058:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.103061:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.103063:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.103067:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.103070:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.103072:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.103073:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.103079:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100304 00000100:00000040:3.0:1496184943.103085:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.103088:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.103106:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.103108:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3850 x1568861104100304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.103114:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100304:12345-10.37.248.124@o2ib1:8 Request procesed in 116us (227us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.103119:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6071 00000100:00000040:3.0:1496184943.103121:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.103125:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184943.105630:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00020000:3.0:1496184943.105641:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000040:7.0:1496184943.134827:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6050 x1568861104100336 msgsize 520 00000100:00100000:7.0:1496184943.134834:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.134836:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.135562:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.135569:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.135571:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.137177:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.137193:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:0.0:1496184943.168611:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:0.0:1496184943.168634:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.197351:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.197366:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.197368:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.198896:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.210014:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.234039:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.245085:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184943.256223:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2450 x1568861104100480 msgsize 520 00000100:00100000:1.0:1496184943.256231:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184943.256233:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.256253:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100480 00000020:00000040:3.0:1496184943.256257:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.256260:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184943.256285:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6072 00000100:00000040:3.0:1496184943.256287:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.256292:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2450 x1568861104100480/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.256301:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100480:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.256313:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.256326:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.256330:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.256334:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.256349:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.256355:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.256358:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.256366:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.256371:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.256373:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.256374:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.256375:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.256378:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.256384:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.256387:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.256390:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.256392:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.256398:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.256401:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.256403:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.256405:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.256413:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100480 00000100:00000040:3.0:1496184943.256421:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.256424:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.256444:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.256447:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2450 x1568861104100480/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.256453:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100480:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.256458:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6072 00000100:00000040:3.0:1496184943.256460:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.256466:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184943.257345:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.257352:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.257354:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.259163:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.270153:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.294250:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.305296:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.326885:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8850 x1568861104100544 msgsize 520 00000100:00100000:7.0:1496184943.326911:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.326913:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.329806:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.340838:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.362826:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5050 x1568861104100576 msgsize 520 00000100:00100000:7.0:1496184943.362840:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.362841:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.363531:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.363537:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.363539:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.365302:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.376325:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184943.398010:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5450 x1568861104100608 msgsize 520 00000100:00100000:7.0:1496184943.398017:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.398018:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.398702:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.398709:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.398718:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.400312:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.411324:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.435450:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.446500:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:2.0:1496184943.457587:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee4050 x1568861104100672 msgsize 520 00000100:00100000:2.0:1496184943.457596:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184943.457597:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.457617:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100672 00000020:00000040:3.0:1496184943.457621:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.457625:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184943.457650:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6073 00000100:00000040:3.0:1496184943.457652:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184943.457658:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee4050 x1568861104100672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.457666:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104100672:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.457679:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.457692:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.457697:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.457700:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.457709:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.457712:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.457714:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.457729:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.457737:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.457739:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.457740:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.457741:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.457744:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.457750:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.457752:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.457755:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.457757:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.457763:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.457765:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.457768:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.457770:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.457778:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100672 00000100:00000040:3.0:1496184943.457785:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.457789:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.457809:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.457812:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee4050 x1568861104100672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.457818:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100672:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.457823:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6073 00000100:00000040:3.0:1496184943.457825:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.457831:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184943.470358:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.481352:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.505539:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.505564:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.536683:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.536707:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.568261:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.568286:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.598537:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.609665:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.631803:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.631810:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.631819:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.633475:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.644536:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184943.655655:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2c50 x1568861104100864 msgsize 520 00000100:00100000:1.0:1496184943.655663:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184943.655665:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.655685:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104100864 00000020:00000040:3.0:1496184943.655689:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.655692:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184943.655717:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6074 00000100:00000040:3.0:1496184943.655719:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184943.655724:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2c50 x1568861104100864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.655733:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104100864:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.655746:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.655758:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.655763:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.655766:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.655782:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.655785:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.655788:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.655796:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.655801:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.655803:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.655804:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.655805:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.655811:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.655817:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.655819:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.655822:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.655824:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.655830:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.655833:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.655835:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.655837:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.655845:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104100864 00000100:00000040:3.0:1496184943.655853:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.655856:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.655876:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.655879:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2c50 x1568861104100864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.655885:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104100864:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (229us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.655899:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6074 00000100:00000040:3.0:1496184943.655901:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.655907:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184943.656786:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.656793:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.656795:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.658440:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.669454:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184943.692426:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.692434:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.692435:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.694157:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.705157:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.726905:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3450 x1568861104100928 msgsize 520 00000100:00100000:7.0:1496184943.726912:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.726913:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.729871:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.740908:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.762727:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2450 x1568861104100960 msgsize 520 00000100:00100000:7.0:1496184943.762734:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.762735:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.763451:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.763458:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.763460:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.765175:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.776197:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184943.798031:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2c50 x1568861104100992 msgsize 520 00000100:00100000:7.0:1496184943.798038:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.798039:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.798764:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.798771:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.798773:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.800706:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.811745:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.835947:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.846975:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:1.0:1496184943.858059:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee3050 x1568861104101056 msgsize 520 00000100:00100000:1.0:1496184943.858068:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184943.858070:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184943.858090:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101056 00000020:00000040:3.0:1496184943.858094:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184943.858097:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184943.858122:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6075 00000100:00000040:3.0:1496184943.858125:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.858130:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee3050 x1568861104101056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184963 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184943.858139:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104101056:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184943.858151:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184943.858163:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184943.858167:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184943.858171:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184943 last 1496184943 00000001:00000040:3.0:1496184943.858180:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184943.858183:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184943.858185:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184943.858193:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184943.858205:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.858208:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.858209:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184943.858210:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184943.858213:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.858218:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.858221:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184943.858224:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.858229:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184943.858234:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184943.858237:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184943.858240:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184943.858241:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184943.858250:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101056 00000100:00000040:3.0:1496184943.858258:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.858262:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.858281:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184943.858284:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee3050 x1568861104101056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184963 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184943.858290:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101056:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (230us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184943.858296:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6075 00000100:00000040:3.0:1496184943.858298:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184943.858304:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184943.870078:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.881076:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184943.902752:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1c50 x1568861104101088 msgsize 520 00000100:00100000:7.0:1496184943.902759:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.902760:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.903434:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.903440:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.903442:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.905215:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.905240:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184943.934273:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2050 x1568861104101120 msgsize 520 00000100:00100000:7.0:1496184943.934286:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.934287:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184943.934952:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184943.934959:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184943.934960:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184943.936612:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.936638:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184943.967225:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.967248:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184943.997834:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.008962:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.032765:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.043816:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:1.0:1496184944.054949:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2050 x1568861104101248 msgsize 520 00000100:00100000:1.0:1496184944.054958:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184944.054960:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184944.055009:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101248 00000020:00000040:3.0:1496184944.055016:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184944.055020:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:3.0:1496184944.055048:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6076 00000100:00000040:3.0:1496184944.055050:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.055055:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2050 x1568861104101248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184944.055071:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101248:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184944.055084:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184944.055096:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184944.055100:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184944.055104:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:3.0:1496184944.055113:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184944.055116:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184944.055119:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184944.055127:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184944.055131:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.055134:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.055135:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184944.055136:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.055139:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.055144:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.055147:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184944.055150:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.055152:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.055157:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184944.055160:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184944.055163:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.055165:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184944.055174:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101248 00000100:00000040:3.0:1496184944.055181:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.055185:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.055205:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.055208:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2050 x1568861104101248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184944.055214:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101248:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (264us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184944.055219:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6076 00000100:00000040:3.0:1496184944.055221:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.055226:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184944.057663:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.068666:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184944.093702:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.093716:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.093717:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.095419:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.106451:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184944.128511:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1050 x1568861104101312 msgsize 520 00000100:00100000:7.0:1496184944.128518:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.128519:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.129201:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.129208:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.129209:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.130953:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.142008:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184944.164109:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0c50 x1568861104101344 msgsize 520 00000100:00100000:7.0:1496184944.164116:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.164121:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.164778:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.164785:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.164786:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.166631:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.177659:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184944.200557:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.200564:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.200573:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.202341:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.213366:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:7.0:1496184944.235154:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104101408 msgsize 520 00000100:00100000:7.0:1496184944.235160:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.235167:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.237578:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.248594:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:1.0:1496184944.259635:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1850 x1568861104101440 msgsize 520 00000100:00100000:1.0:1496184944.259643:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184944.259645:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184944.259674:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101440 00000020:00000040:3.0:1496184944.259678:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184944.259681:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184944.259706:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6077 00000100:00000040:3.0:1496184944.259708:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184944.259716:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1850 x1568861104101440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184944.259726:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104101440:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184944.259738:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184944.259750:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184944.259755:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184944.259759:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:3.0:1496184944.259767:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184944.259770:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184944.259773:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184944.259781:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184944.259785:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.259787:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.259788:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184944.259789:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.259792:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.259798:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.259800:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184944.259811:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.259813:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.259819:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184944.259822:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184944.259824:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.259826:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184944.259834:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101440 00000100:00000040:3.0:1496184944.259841:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.259846:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.259866:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.259869:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1850 x1568861104101440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184944.259876:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101440:12345-10.37.248.124@o2ib1:8 Request procesed in 151us (240us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184944.259881:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6077 00000100:00000040:3.0:1496184944.259883:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.259889:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184944.270054:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.281058:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.304103:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.304127:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.334711:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00020000:3.0:1496184944.334735:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000040:2.0:1496184944.341907:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184944.341931:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000040:2.0:1496184944.341950:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.341961:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f76aa00] to pc [ptlrpcd_00_00:0] req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.341969:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1496184944.341972:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000040:2.0:1496184944.341976:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184944.341980:0:13690:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00000040:2.0:1496184944.341982:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f768f00] to pc [ptlrpcd_00_01:1] req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.341989:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1496184944.341991:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000020:00000040:2.0:1496184944.341992:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:2.0:1496184944.341995:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.342000:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f769200] to pc [ptlrpcd_00_02:2] req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184944.342001:0:13689:0:(ptlrpcd.c:409:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1496184944.342003:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:d86c02e9-1939-02b8-4331-d3612fcf0106:13690:1568861159710240:10.37.248.196@o2ib1:400 00000100:00000040:2.0:1496184944.342009:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 5000 (4297383370) 00000100:00000040:1.0:1496184944.342010:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184944.342019:0:13689:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710256:10.37.248.197@o2ib1:400 00000100:00000040:0.0:1496184944.342021:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184944.342027:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000040:3.0:1496184944.342028:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000040:1.0:1496184944.342031:0:13689:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184944.342038:0:13691:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710272:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184944.342040:0:13689:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:3.0:1496184944.342059:0:13691:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184944.342065:0:13691:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:2.0:1496184944.342073:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184944.342095:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.342102:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.342353:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184944.342378:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184944.342387:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:3.0:1496184944.342410:0:13691:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00000100:00000040:3.0:1496184944.342413:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184944.342416:0:13689:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000040:3.0:1496184944.342419:0:13691:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:1.0:1496184944.342421:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00100000:3.0:1496184944.342424:0:13691:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_02:lustre-MDT0000-lwp-OST0001_UUID:13691:1568861159710272:10.37.248.197@o2ib1:400 00000100:00000040:1.0:1496184944.342429:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000040:3.0:1496184944.342430:0:13691:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f769200 x1568861159710272/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000040:0.0:1496184944.342433:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00100000:1.0:1496184944.342434:0:13689:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710256:10.37.248.197@o2ib1:400 00000020:00000040:3.0:1496184944.342436:0:13691:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000100:00000040:1.0:1496184944.342438:0:13689:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f768f00 x1568861159710256/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000040:0.0:1496184944.342441:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Interpret:RN/0/0 rc 0/0 00000020:00000040:1.0:1496184944.342443:0:13689:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000100:00100000:0.0:1496184944.342447:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:d86c02e9-1939-02b8-4331-d3612fcf0106:13690:1568861159710240:10.37.248.196@o2ib1:400 00000100:00000040:0.0:1496184944.342452:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f76aa00 x1568861159710240/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184951 ref 1 fl Complete:RN/0/0 rc 0/0 00000020:00000040:0.0:1496184944.342458:0:13690:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000100:00020000:3.0:1496184944.364834:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00020000:3.0:1496184944.364860:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000040:7.0:1496184944.391950:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9850 x1568861104101568 msgsize 520 00000100:00100000:7.0:1496184944.391957:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.391958:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.392645:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.392652:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.392653:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.394495:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184944.405619:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184944.429569:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184944.440623:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:2.0:1496184944.451716:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee2850 x1568861104101632 msgsize 520 00000100:00100000:2.0:1496184944.451734:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184944.451735:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184944.451756:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101632 00000020:00000040:3.0:1496184944.451759:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184944.451763:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184944.451787:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6078 00000100:00000040:3.0:1496184944.451790:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184944.451795:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee2850 x1568861104101632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184944.451804:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104101632:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184944.451816:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184944.451829:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184944.451834:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184944.451838:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:3.0:1496184944.451846:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184944.451849:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184944.451851:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184944.451860:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184944.451864:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.451874:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.451877:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184944.451879:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.451882:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.451887:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.451898:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184944.451901:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.451904:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.451909:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184944.451912:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184944.451914:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.451916:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184944.451925:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101632 00000100:00000040:3.0:1496184944.451932:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.451936:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.451955:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.451965:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee2850 x1568861104101632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184944.451971:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101632:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (255us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184944.451977:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6078 00000100:00000040:3.0:1496184944.451979:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.451985:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184944.452925:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.452932:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.452934:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.454823:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184944.465831:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184944.494155:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184944.505191:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184944.527060:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdac50 x1568861104101696 msgsize 520 00000100:00100000:7.0:1496184944.527068:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.527069:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.529763:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184944.540809:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184944.565225:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184944.576268:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184944.600330:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184944.611352:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184944.633400:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.633414:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.633415:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.635140:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184944.646172:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:1.0:1496184944.657263:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1450 x1568861104101840 msgsize 520 00000100:00100000:1.0:1496184944.657272:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184944.657273:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184944.657293:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101840 00000020:00000040:3.0:1496184944.657297:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184944.657301:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184944.657326:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6079 00000100:00000040:3.0:1496184944.657329:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.657334:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1450 x1568861104101840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184944.657343:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104101840:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184944.657355:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184944.657367:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184944.657372:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184944.657375:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:3.0:1496184944.657391:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184944.657394:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184944.657397:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184944.657405:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184944.657409:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.657411:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.657412:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184944.657413:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184944.657416:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.657422:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.657427:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184944.657431:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.657433:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184944.657438:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184944.657441:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184944.657443:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184944.657445:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184944.657453:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101840 00000100:00000040:3.0:1496184944.657461:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.657465:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.657486:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184944.657488:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1450 x1568861104101840/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184944.657494:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101840:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (231us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184944.657500:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6079 00000100:00000040:3.0:1496184944.657502:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184944.657507:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184944.658400:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600dc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.658407:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.658408:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.670070:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184944.681075:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184944.703018:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb450 x1568861104101872 msgsize 520 00000100:00100000:7.0:1496184944.703025:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.703027:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.703715:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.703722:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.703724:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.705493:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184944.705518:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:3.0:1496184944.733987:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1c50 x1568861104101904 msgsize 520 00000100:00100000:3.0:1496184944.733995:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184944.733997:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184944.734044:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104101904 00000020:00000040:0.0:1496184944.734048:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184944.734051:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184944.734076:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6080 00000100:00000040:0.0:1496184944.734078:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184944.734082:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1c50 x1568861104101904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184944.734091:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101904:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184944.734102:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184944.734112:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184944.734116:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184944.734120:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:0.0:1496184944.734127:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184944.734129:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184944.734131:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184944.734137:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184944.734141:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184944.734143:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184944.734144:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184944.734145:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184944.734147:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184944.734153:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184944.734155:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184944.734160:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184944.734162:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184944.734167:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184944.734169:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184944.734171:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184944.734173:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184944.734180:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104101904 00000100:00000040:0.0:1496184944.734186:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184944.734189:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184944.734205:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184944.734208:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1c50 x1568861104101904/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184944.734214:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104101904:12345-10.37.248.124@o2ib1:8 Request procesed in 124us (226us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184944.734219:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6080 00000100:00000040:0.0:1496184944.734221:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184944.734225:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184944.734710:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ec50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.734718:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.734719:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.736334:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184944.736357:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:7.0:1496184944.764635:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc050 x1568861104101936 msgsize 520 00000100:00100000:7.0:1496184944.764642:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.764643:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.766912:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184944.766937:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:7.0:1496184944.795192:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc450 x1568861104101968 msgsize 520 00000100:00100000:7.0:1496184944.795199:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.795209:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.795872:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.795879:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.795880:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.797565:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184944.808711:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184944.832624:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184944.843694:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:1.0:1496184944.854826:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee1050 x1568861104102032 msgsize 520 00000100:00100000:1.0:1496184944.854835:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184944.854837:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184944.854862:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104102032 00000020:00000040:2.0:1496184944.854866:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184944.854870:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184944.854904:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6081 00000100:00000040:2.0:1496184944.854907:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184944.854912:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee1050 x1568861104102032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184964 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184944.854921:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102032:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184944.854933:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184944.854948:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184944.854953:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184944.854957:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184944 last 1496184943 00000001:00000040:2.0:1496184944.854972:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184944.854975:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184944.854978:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184944.854986:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184944.854991:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184944.854993:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184944.854994:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184944.854995:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184944.854998:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184944.855004:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184944.855006:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184944.855010:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184944.855012:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184944.855017:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184944.855020:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184944.855022:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184944.855024:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184944.855032:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104102032 00000100:00000040:2.0:1496184944.855039:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184944.855043:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184944.855063:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184944.855066:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee1050 x1568861104102032/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184964 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184944.855072:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102032:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (245us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184944.855078:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6081 00000100:00000040:2.0:1496184944.855080:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184944.855086:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184944.857679:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184944.868691:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184944.895426:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184944.906442:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184944.930759:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184944.941819:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184944.963618:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd450 x1568861104102128 msgsize 520 00000100:00100000:7.0:1496184944.963625:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.963626:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184944.964357:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184944.964364:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.964370:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184944.966197:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184944.977239:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184944.999009:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd850 x1568861104102160 msgsize 520 00000100:00100000:7.0:1496184944.999016:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184944.999025:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.001287:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184945.012313:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:7.0:1496184945.033711:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bddc50 x1568861104102192 msgsize 520 00000100:00100000:7.0:1496184945.033718:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.033723:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.036019:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184945.047086:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:1.0:1496184945.058181:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0450 x1568861104102224 msgsize 520 00000100:00100000:1.0:1496184945.058190:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184945.058192:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184945.058239:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104102224 00000020:00000040:2.0:1496184945.058244:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184945.058248:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184945.058276:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6082 00000100:00000040:2.0:1496184945.058278:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184945.058284:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0450 x1568861104102224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184965 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184945.058292:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102224:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184945.058312:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184945.058324:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184945.058329:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184945.058333:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184945 last 1496184945 00000001:00000040:2.0:1496184945.058341:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184945.058345:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184945.058347:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184945.058356:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184945.058360:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184945.058362:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184945.058363:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184945.058364:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184945.058368:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.058373:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184945.058376:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184945.058382:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.058384:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184945.058390:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184945.058393:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184945.058395:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184945.058397:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184945.058405:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104102224 00000100:00000040:2.0:1496184945.058413:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.058416:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.058436:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.058439:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0450 x1568861104102224/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184965 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184945.058445:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102224:12345-10.37.248.124@o2ib1:8 Request procesed in 154us (264us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184945.058451:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6082 00000100:00000040:2.0:1496184945.058453:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184945.058458:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184945.059309:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.059316:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.059317:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.069879:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184945.080880:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:1.0:1496184945.102537:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0850 x1568861104102256 msgsize 520 00000100:00100000:1.0:1496184945.102545:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184945.102546:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184945.102562:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104102256 00000020:00000040:2.0:1496184945.102566:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184945.102569:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:2.0:1496184945.102588:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6083 00000100:00000040:2.0:1496184945.102590:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184945.102594:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0850 x1568861104102256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184965 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184945.102602:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102256:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184945.102611:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184945.102621:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184945.102624:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184945.102629:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184945 last 1496184945 00000001:00000040:2.0:1496184945.102635:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184945.102638:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184945.102639:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184945.102646:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184945.102650:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184945.102651:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184945.102652:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184945.102653:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184945.102656:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.102660:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184945.102663:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184945.102665:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.102667:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184945.102671:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184945.102674:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184945.102676:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184945.102678:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184945.102684:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104102256 00000100:00000040:2.0:1496184945.102690:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.102693:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.102714:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184945.102716:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0850 x1568861104102256/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184965 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184945.102722:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102256:12345-10.37.248.124@o2ib1:8 Request procesed in 121us (185us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184945.102730:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6083 00000100:00000040:2.0:1496184945.102732:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184945.102737:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184945.103233:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.103240:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.103241:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.105218:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00020000:3.0:1496184945.105244:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000040:7.0:1496184945.134223:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde450 x1568861104102288 msgsize 520 00000100:00100000:7.0:1496184945.134230:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.134231:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.134886:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.134904:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.134905:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.136565:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00020000:3.0:1496184945.136590:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00000040:7.0:1496184945.164551:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde050 x1568861104102320 msgsize 520 00000100:00100000:7.0:1496184945.164558:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.164560:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.165149:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.165156:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.165157:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.166866:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184945.166901:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:7.0:1496184945.194365:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde850 x1568861104102352 msgsize 520 00000100:00100000:7.0:1496184945.194372:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.194373:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.194984:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.194996:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.194997:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.196575:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184945.207714:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:7.0:1496184945.229492:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdec50 x1568861104102384 msgsize 520 00000100:00100000:7.0:1496184945.229499:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.229500:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.230141:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.230148:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.230149:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.231600:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184945.242668:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:1.0:1496184945.253740:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0c50 x1568861104102416 msgsize 520 00000100:00100000:1.0:1496184945.253749:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184945.253751:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184945.253773:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104102416 00000020:00000040:3.0:1496184945.253777:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184945.253789:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184945.253814:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6084 00000100:00000040:3.0:1496184945.253820:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184945.253825:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0c50 x1568861104102416/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184965 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184945.253833:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+10:12082:x1568861104102416:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184945.253845:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184945.253858:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00010000:02000400:3.0:1496184945.253862:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184945.253866:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184945 last 1496184945 00000001:00000040:3.0:1496184945.253875:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184945.253878:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184945.253881:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184945.253900:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184945.253905:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184945.253907:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184945.253908:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184945.253909:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184945.253913:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.253918:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184945.253921:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184945.253924:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.253926:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184945.253931:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184945.253934:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184945.253936:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184945.253938:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184945.253953:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104102416 00000100:00000040:3.0:1496184945.253961:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.253965:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.253985:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.253987:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0c50 x1568861104102416/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184965 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184945.253993:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102416:12345-10.37.248.124@o2ib1:8 Request procesed in 161us (253us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184945.253998:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6084 00000100:00000040:3.0:1496184945.254001:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184945.254006:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184945.256614:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184945.267609:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:7.0:1496184945.295216:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.295222:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.295224:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.297201:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184945.297212:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184945.327961:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184945.327986:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:7.0:1496184945.356421:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf850 x1568861104102512 msgsize 520 00000100:00100000:7.0:1496184945.356428:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.356430:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.357043:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.357050:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.357052:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.358592:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.358618:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:7.0:1496184945.387380:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c9050 x1568861104102544 msgsize 520 00000100:00100000:7.0:1496184945.387387:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.387389:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.388089:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.388096:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.388097:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.389774:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184945.389798:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:0.0:1496184945.420972:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:0.0:1496184945.432093:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184945.447797:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.458836:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.483066:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184945.494135:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:7.0:1496184945.516852:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.516859:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.516860:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.518529:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184945.529604:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:7.0:1496184945.551926:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.551933:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.551934:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.553636:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184945.564682:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184945.588415:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184945.599462:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184945.623615:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184945.634649:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:7.0:1496184945.656140:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6850 x1568861104102800 msgsize 520 00000100:00100000:7.0:1496184945.656147:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.656153:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.656904:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.656911:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.656912:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.658486:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184945.669530:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:7.0:1496184945.691260:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6050 x1568861104102832 msgsize 520 00000100:00100000:7.0:1496184945.691267:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.691269:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.693606:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184945.704630:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000040:7.0:1496184945.726548:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5c50 x1568861104102864 msgsize 520 00000100:00100000:7.0:1496184945.726555:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.726557:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.727270:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.727277:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.727283:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.729235:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184945.740280:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184945.764730:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184945.775769:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184945.799805:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184945.810841:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184945.834923:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.845944:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:1.0:1496184945.857044:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f850 x1568861104102992 msgsize 520 00000100:00100000:1.0:1496184945.857053:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184945.857055:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184945.857078:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104102992 00000020:00000040:3.0:1496184945.857082:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184945.857085:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:3.0:1496184945.857109:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6085 00000100:00000040:3.0:1496184945.857112:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184945.857117:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f850 x1568861104102992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184965 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184945.857126:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102992:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184945.857139:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184945.857151:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184945.857155:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184945.857159:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184945 last 1496184945 00000001:00000040:3.0:1496184945.857175:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184945.857178:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184945.857180:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184945.857191:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184945.857196:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184945.857198:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184945.857199:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184945.857200:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184945.857204:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.857210:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184945.857212:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184945.857215:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.857218:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184945.857223:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184945.857226:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184945.857229:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184945.857230:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184945.857239:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104102992 00000100:00000040:3.0:1496184945.857246:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.857250:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.857269:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184945.857272:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f850 x1568861104102992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184965 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184945.857278:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104102992:12345-10.37.248.124@o2ib1:8 Request procesed in 153us (233us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184945.857284:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6085 00000100:00000040:3.0:1496184945.857285:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184945.857291:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184945.870236:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.881250:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00000040:3.0:1496184945.903091:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee0050 x1568861104103024 msgsize 520 00000100:00100000:3.0:1496184945.903099:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184945.903101:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:0.0:1496184945.903146:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104103024 00000020:00000040:0.0:1496184945.903150:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:0.0:1496184945.903153:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000100:00100000:0.0:1496184945.903174:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6086 00000100:00000040:0.0:1496184945.903176:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184945.903179:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee0050 x1568861104103024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184965 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184945.903187:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103024:12345-10.37.248.124@o2ib1:8 00000020:00000040:0.0:1496184945.903198:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:0.0:1496184945.903209:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:0.0:1496184945.903212:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:0.0:1496184945.903215:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184945 last 1496184945 00000001:00000040:0.0:1496184945.903222:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:0.0:1496184945.903225:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:0.0:1496184945.903226:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:0.0:1496184945.903232:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:0.0:1496184945.903236:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184945.903237:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184945.903238:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184945.903240:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:0.0:1496184945.903242:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184945.903247:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184945.903249:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:0.0:1496184945.903252:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184945.903254:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184945.903258:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:0.0:1496184945.903260:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:0.0:1496184945.903262:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184945.903263:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:0.0:1496184945.903269:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104103024 00000100:00000040:0.0:1496184945.903274:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184945.903278:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:0.0:1496184945.903299:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:0.0:1496184945.903302:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee0050 x1568861104103024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184965 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184945.903308:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103024:12345-10.37.248.124@o2ib1:8 Request procesed in 121us (216us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184945.903314:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6086 00000100:00000040:0.0:1496184945.903315:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184945.903320:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184945.905661:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.905685:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f600 00000100:00020000:3.0:1496184945.936869:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00020000:3.0:1496184945.936902:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000040:2.0:1496184945.949716:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee7450 x1568861152397680 msgsize 224 00000100:00100000:2.0:1496184945.949723:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184945.949725:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:1.0:1496184945.949768:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397680 00000020:00000040:1.0:1496184945.949771:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000040:1.0:1496184945.949773:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000100:00000040:1.0:1496184945.949781:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00100000:1.0:1496184945.949794:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 958 00000100:00000040:1.0:1496184945.949796:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000040:1.0:1496184945.949800:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee7450 x1568861152397680/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:696/0 lens 224/0 e 0 to 0 dl 1496184951 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00100000:1.0:1496184945.949807:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397680:12345-10.37.248.197@o2ib1:13 00002000:00000024:1.0:1496184945.949824:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00010000:00000040:1.0:1496184945.949829:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397680 00000100:00000040:1.0:1496184945.949833:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184945.949837:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000040:1.0:1496184945.949851:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000040:1.0:1496184945.949854:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee7450 x1568861152397680/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:696/0 lens 224/336 e 0 to 0 dl 1496184951 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00100000:1.0:1496184945.949860:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397680:12345-10.37.248.197@o2ib1:13 Request procesed in 53us (143us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184945.949865:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 958 00000100:00000040:1.0:1496184945.949866:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000020:00000040:1.0:1496184945.949871:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 00000100:00000040:7.0:1496184945.966072:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.966079:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.966080:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.967623:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184945.967648:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:7.0:1496184945.995872:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5450 x1568861104103120 msgsize 520 00000100:00100000:7.0:1496184945.995879:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.995880:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184945.996585:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184945.996592:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184945.996593:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184945.998154:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184946.009314:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:7.0:1496184946.030914:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4850 x1568861104103152 msgsize 520 00000100:00100000:7.0:1496184946.030921:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.030922:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.033277:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184946.044368:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:1.0:1496184946.055520:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f050 x1568861104103184 msgsize 520 00000100:00100000:1.0:1496184946.055529:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184946.055530:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:2.0:1496184946.055555:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104103184 00000020:00000040:2.0:1496184946.055560:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:2.0:1496184946.055564:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00100000:2.0:1496184946.055592:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6087 00000100:00000040:2.0:1496184946.055594:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184946.055599:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f050 x1568861104103184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184966 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:2.0:1496184946.055608:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103184:12345-10.37.248.124@o2ib1:8 00000020:00000040:2.0:1496184946.055621:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:2.0:1496184946.055633:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:2.0:1496184946.055638:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:2.0:1496184946.055641:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184946 last 1496184945 00000001:00000040:2.0:1496184946.055657:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:2.0:1496184946.055661:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:2.0:1496184946.055663:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:2.0:1496184946.055671:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:2.0:1496184946.055676:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184946.055678:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184946.055679:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184946.055680:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:2.0:1496184946.055686:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184946.055691:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184946.055694:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:2.0:1496184946.055697:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184946.055699:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184946.055704:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:2.0:1496184946.055707:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:2.0:1496184946.055710:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184946.055712:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:2.0:1496184946.055720:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104103184 00000100:00000040:2.0:1496184946.055727:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184946.055730:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:2.0:1496184946.055750:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:2.0:1496184946.055753:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f050 x1568861104103184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184966 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184946.055760:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103184:12345-10.37.248.124@o2ib1:8 Request procesed in 152us (239us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184946.055765:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6087 00000100:00000040:2.0:1496184946.055767:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184946.055772:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000040:7.0:1496184946.056654:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.056662:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.056671:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.058280:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184946.069309:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:7.0:1496184946.092655:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.092662:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.092664:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.094494:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184946.105539:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184946.130041:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184946.141112:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184946.162946:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4450 x1568861104103280 msgsize 520 00000100:00100000:7.0:1496184946.162952:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.162954:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.163691:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.163698:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.163705:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.165731:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184946.176786:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184946.198941:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3050 x1568861104103312 msgsize 520 00000100:00100000:7.0:1496184946.198947:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.198949:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.199700:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.199707:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.199708:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.201722:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184946.212776:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184946.231407:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3c50 x1568861104103344 msgsize 520 00000100:00100000:7.0:1496184946.231414:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.231415:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.232114:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.232127:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.232128:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.233731:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184946.244772:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:2.0:1496184946.255847:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8fc50 x1568861104103376 msgsize 520 00000100:00100000:2.0:1496184946.255864:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184946.255866:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00100000:3.0:1496184946.255887:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104103376 00000020:00000040:3.0:1496184946.255900:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:3.0:1496184946.255903:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00100000:3.0:1496184946.255929:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6088 00000100:00000040:3.0:1496184946.255932:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000040:3.0:1496184946.255937:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8fc50 x1568861104103376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184966 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:3.0:1496184946.255949:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104103376:12345-10.37.248.124@o2ib1:8 00000020:00000040:3.0:1496184946.255961:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000020:00000040:3.0:1496184946.255974:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:02000400:3.0:1496184946.255979:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00080000:3.0:1496184946.255982:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184946 last 1496184945 00000001:00000040:3.0:1496184946.255991:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000040:3.0:1496184946.255994:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00002000:00100000:3.0:1496184946.255996:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00100000:3.0:1496184946.256017:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00000100:00000040:3.0:1496184946.256022:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184946.256024:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184946.256025:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184946.256026:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000040:3.0:1496184946.256029:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184946.256035:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184946.256038:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000040:3.0:1496184946.256041:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184946.256043:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184946.256048:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:02000000:3.0:1496184946.256051:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000020:00000040:3.0:1496184946.256054:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184946.256056:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000040:3.0:1496184946.256064:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104103376 00000100:00000040:3.0:1496184946.256072:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184946.256075:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000040:3.0:1496184946.256095:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000040:3.0:1496184946.256098:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8fc50 x1568861104103376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184966 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184946.256104:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103376:12345-10.37.248.124@o2ib1:8 Request procesed in 156us (257us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184946.256110:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6088 00000100:00000040:3.0:1496184946.256112:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184946.256117:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00020000:3.0:1496184946.258675:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184946.269688:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:7.0:1496184946.291515:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3850 x1568861104103408 msgsize 520 00000100:00100000:7.0:1496184946.291527:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.291528:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.294398:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184946.305442:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000040:7.0:1496184946.326745:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3450 x1568861104103440 msgsize 520 00000100:00100000:7.0:1496184946.326752:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.326753:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.329202:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184946.340232:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:7.0:1496184946.357356:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2450 x1568861104103472 msgsize 520 00000100:00100000:7.0:1496184946.357363:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.357364:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.358035:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ac50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.358042:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.358043:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.359772:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184946.370815:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000040:7.0:1496184946.391064:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2c50 x1568861104103504 msgsize 520 00000100:00100000:7.0:1496184946.391071:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.391072:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.391794:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.391801:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.391802:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.393667:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184946.404707:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00020000:3.0:1496184946.424639:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00020000:3.0:1496184946.435664:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000040:7.0:1496184946.452679:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1c50 x1568861104103568 msgsize 520 00000100:00100000:7.0:1496184946.452686:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.452687:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.453466:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.453473:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.453478:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.455432:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00020000:3.0:1496184946.466447:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000040:7.0:1496184946.483191:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2050 x1568861104103600 msgsize 520 00000100:00100000:7.0:1496184946.483198:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.483199:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.483955:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.483962:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.483969:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.485826:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184946.496848:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00020000:3.0:1496184946.516141:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00020000:3.0:1496184946.527177:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000040:7.0:1496184946.549620:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.549633:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.549635:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.551578:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00020000:3.0:1496184946.562629:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000040:7.0:1496184946.584098:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1450 x1568861104103696 msgsize 520 00000100:00100000:7.0:1496184946.584105:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.584106:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.584758:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.584765:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.584766:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.586491:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00020000:3.0:1496184946.597527:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000040:7.0:1496184946.619481:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0850 x1568861104103728 msgsize 520 00000100:00100000:7.0:1496184946.619488:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.619489:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.620243:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600cc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.620250:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.620259:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.622130:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00020000:3.0:1496184946.633165:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000040:7.0:1496184946.654766:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1050 x1568861104103760 msgsize 520 00000100:00100000:7.0:1496184946.654773:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.654779:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.655527:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.655534:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.655539:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.657274:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00020000:3.0:1496184946.668307:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000040:7.0:1496184946.689995:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0c50 x1568861104103792 msgsize 520 00000100:00100000:7.0:1496184946.690002:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.690003:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.690699:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.690706:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.690708:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.692535:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00020000:3.0:1496184946.703594:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000040:7.0:1496184946.725676:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd8c50 x1568861104103824 msgsize 520 00000100:00100000:7.0:1496184946.725683:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.725684:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000040:7.0:1496184946.726417:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.726423:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.726427:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00020000:3.0:1496184946.728120:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00020000:3.0:1496184946.739166:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000001:7.0:1496184946.760736:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.760738:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184946.760741:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104103856 msgsize 520 00000100:00100000:7.0:1496184946.760746:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.760747:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.760749:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:7.0:1496184946.760833:0:16457:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:7.0:1496184946.760835:0:16457:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:7.0:1496184946.760838:0:16457:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:7.0:1496184946.760839:0:16457:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:7.0:1496184946.760841:0:16457:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184946.760843:0:16457:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:7.0:1496184946.760845:0:16457:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:7.0:1496184946.760846:0:16457:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184946.761082:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184946.761084:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184946.761086:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77e400. 02000000:00000001:7.0:1496184946.761089:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184946.761090:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184946.761518:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.761520:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184946.761523:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.761530:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.761532:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.761535:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:7.0:1496184946.761633:0:13742:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:7.0:1496184946.761635:0:13742:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:7.0:1496184946.761637:0:13742:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:7.0:1496184946.761638:0:13742:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:7.0:1496184946.761639:0:13742:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184946.761640:0:13742:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:7.0:1496184946.761642:0:13742:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:7.0:1496184946.761643:0:13742:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184946.763554:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.763558:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184946.774656:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.774661:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.774662:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184946.785670:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184946.796155:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.796169:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184946.796171:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104103888 msgsize 520 00000100:00100000:7.0:1496184946.796176:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.796177:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.796179:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:7.0:1496184946.796233:0:16457:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:7.0:1496184946.796235:0:16457:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:7.0:1496184946.796237:0:16457:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:7.0:1496184946.796238:0:16457:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:7.0:1496184946.796239:0:16457:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184946.796240:0:16457:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:7.0:1496184946.796243:0:16457:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:7.0:1496184946.796244:0:16457:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184946.796926:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.796927:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184946.796929:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600dc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.796943:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.796944:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.796947:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:7.0:1496184946.797081:0:13742:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:7.0:1496184946.797082:0:13742:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:7.0:1496184946.797083:0:13742:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:7.0:1496184946.797084:0:13742:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:7.0:1496184946.797088:0:13742:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184946.797089:0:13742:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:7.0:1496184946.797091:0:13742:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:7.0:1496184946.797091:0:13742:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184946.799106:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.799109:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184946.810151:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.810156:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.810157:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184946.821155:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184946.831951:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.831952:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184946.831955:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9050 x1568861104103920 msgsize 520 00000100:00100000:7.0:1496184946.831960:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.831961:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.831963:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184946.834516:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.834520:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.845577:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.845585:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.845586:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.856630:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184946.856717:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184946.856719:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184946.856722:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ec50 x1568861104103952 msgsize 520 00000100:00100000:1.0:1496184946.856729:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184946.856731:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184946.856734:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184946.856740:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184946.856742:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184946.856746:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184946.856749:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184946.856751:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184946.856752:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184946.856756:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184946.856760:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104103952 00000020:00000001:3.0:1496184946.856763:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184946.856764:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184946.856767:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184946.856772:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:00000001:3.0:1496184946.856774:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184946.856776:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184946.856780:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184946.856781:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184946.856785:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100fc65e00. 00000020:00000010:3.0:1496184946.856788:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132fb00. 00000020:00000010:3.0:1496184946.856791:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00000100:00000001:3.0:1496184946.856793:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184946.856794:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184946.856796:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184946.856801:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184946.856806:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8ec50 x1568861104103952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184966 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184946.856812:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184946.856813:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.856817:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184946.856821:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184946.856822:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184946.856823:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184946.856824:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184946.856825:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184946.856827:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184946.856830:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6089 00000100:00000040:3.0:1496184946.856832:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184946.856834:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736407632 : -131906973143984 : ffff880803b8ec50) 00000100:00000040:3.0:1496184946.856837:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ec50 x1568861104103952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184966 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184946.856842:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184946.856843:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184946.856845:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103952:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184946.856847:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104103952 00000020:00000001:3.0:1496184946.856850:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184946.856855:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184946.856863:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184946.856868:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184946.856869:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184946.856872:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fac00. 02000000:00000001:3.0:1496184946.856874:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.856875:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184946.856889:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184946.856898:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184946.856901:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184946.856903:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184946.856905:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184946.856908:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184946 last 1496184945 00010000:00000001:3.0:1496184946.856911:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184946.856914:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184946.856916:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184946.856918:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184946.856920:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184946.856922:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184946.856923:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184946.856925:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184946.856927:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184946.856929:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184946.856930:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184946.856932:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184946.856933:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184946.856938:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184946.856940:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184946.856941:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184946.856944:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184946.856945:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184946.856947:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184946.856950:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184946.856951:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184946.856952:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184946.856955:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184946.856956:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184946.856958:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184946.856960:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184946.856961:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184946.856963:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184946.856965:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184946.856966:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184946.856967:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184946.856969:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184946.856971:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184946.856973:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184946.856973:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184946.856976:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184946.856978:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184946.856979:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.856981:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184946.856985:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184946.856987:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184946.856990:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184946.856992:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184946.856994:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184946.856995:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.856996:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184946.856997:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184946.856999:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184946.857001:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184946.857003:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184946.857004:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.857007:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184946.857009:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184946.857010:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184946.857012:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184946.857014:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184946.857015:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184946.857017:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184946.857019:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184946.857020:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184946.857022:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184946.857024:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184946.857027:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184946.857029:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184946.857030:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184946.857031:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184946.857032:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184946.857034:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184946.857035:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184946.857037:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184946.857039:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184946.857041:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104103952 00010000:00000001:3.0:1496184946.857043:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184946.857054:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8ec50 x1568861104103952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184966 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184946.857061:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184946.857063:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184946.857064:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=100 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184946.857067:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184946.857069:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184946.857070:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184946.857072:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184946.857073:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184946.857074:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.857076:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184946.857079:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184946.857082:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104103952, offset 0 00000100:00000001:3.0:1496184946.857099:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184946.857100:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184946.857102:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184946.857103:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184946.857104:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184946.857105:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184946.857106:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184946.857108:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ec50 x1568861104103952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184966 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184946.857114:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104103952:12345-10.37.248.124@o2ib1:8 Request procesed in 270us (390us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184946.857119:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6089 00000100:00000040:3.0:1496184946.857121:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184946.857122:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184946.857123:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184946.857126:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132fb00. 00000020:00000010:3.0:1496184946.857128:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184946.857138:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100fc65e00. 00000020:00000040:3.0:1496184946.857139:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184946.857141:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184946.857144:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184946.857149:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184946.857150:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184946.869965:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.869967:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.880969:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.880978:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.880979:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.891973:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184946.903462:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184946.903464:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184946.903465:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779400. 02000000:00000001:7.0:1496184946.903468:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184946.903469:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184946.903798:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.903800:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184946.903803:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.903808:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.903809:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.903811:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184946.905685:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.905688:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.905708:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.905712:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.905716:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184946.905721:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.937494:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.937497:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184946.937515:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184946.937519:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.937520:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184946.937524:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00004000:00000001:1.0:1496184946.951949:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184946.951952:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184946.951955:0:14306:0:(service.c:1448:ptlrpc_at_check_timed()) Process entered 00000100:00000001:1.0:1496184946.951957:0:14306:0:(service.c:1460:ptlrpc_at_check_timed()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:1.0:1496184946.951960:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184946.951961:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:2.0:1496184946.968946:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:2.0:1496184946.968948:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:2.0:1496184946.968954:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:2.0:1496184946.968958:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:2.0:1496184946.968959:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:2.0:1496184946.968963:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184946.997446:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.997448:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184946.997450:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bda450 x1568861104104080 msgsize 520 00000100:00100000:7.0:1496184946.997455:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.997456:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.997459:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184946.997816:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184946.997819:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184946.997821:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779000. 02000000:00000001:7.0:1496184946.997826:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184946.997827:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184946.998232:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184946.998234:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184946.998236:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184946.998241:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184946.998249:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184946.998251:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184946.999876:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184946.999879:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.011051:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.011056:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.011056:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.022070:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.035388:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.035391:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.046459:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.046464:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.046465:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.057532:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184947.057601:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184947.057612:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184947.057615:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e050 x1568861104104144 msgsize 520 00000100:00100000:1.0:1496184947.057622:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184947.057623:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184947.057627:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184947.057633:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.057635:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.057639:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184947.057642:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184947.057644:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184947.057646:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184947.057650:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184947.057654:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104144 00000020:00000001:3.0:1496184947.057657:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184947.057658:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184947.057660:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184947.057663:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184947.057665:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184947.057666:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184947.057671:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.057672:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184947.057674:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88101021d800. 00000020:00000010:3.0:1496184947.057680:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132fe00. 00000020:00000010:3.0:1496184947.057683:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00000100:00000001:3.0:1496184947.057686:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184947.057687:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184947.057688:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184947.057694:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184947.057699:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8e050 x1568861104104144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.057705:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184947.057706:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057710:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184947.057714:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.057715:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184947.057716:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.057717:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.057719:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184947.057720:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184947.057723:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6090 00000100:00000040:3.0:1496184947.057725:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.057727:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736404560 : -131906973147056 : ffff880803b8e050) 00000100:00000040:3.0:1496184947.057730:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e050 x1568861104104144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.057735:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.057736:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184947.057738:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104104144:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184947.057741:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104144 00000020:00000001:3.0:1496184947.057743:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184947.057748:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184947.057755:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184947.057760:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184947.057762:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184947.057764:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fc000. 02000000:00000001:3.0:1496184947.057766:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057768:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.057775:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184947.057777:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184947.057779:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184947.057782:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184947.057783:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184947.057786:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:3.0:1496184947.057797:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184947.057800:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184947.057803:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184947.057805:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184947.057807:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184947.057808:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184947.057810:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184947.057811:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184947.057814:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184947.057815:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184947.057817:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184947.057818:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184947.057820:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184947.057825:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184947.057827:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184947.057828:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184947.057831:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184947.057832:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184947.057834:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184947.057837:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184947.057837:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184947.057839:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184947.057841:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184947.057843:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184947.057845:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.057846:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184947.057848:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184947.057850:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184947.057852:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.057853:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.057854:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184947.057856:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184947.057858:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.057859:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184947.057860:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.057862:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.057864:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.057866:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057867:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184947.057871:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184947.057873:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184947.057876:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184947.057878:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.057879:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.057880:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057882:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.057883:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.057885:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.057887:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184947.057889:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184947.057899:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057900:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184947.057903:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184947.057904:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184947.057906:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184947.057908:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184947.057910:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184947.057911:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184947.057914:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.057915:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.057917:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184947.057919:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.057922:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184947.057924:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184947.057925:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184947.057926:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184947.057928:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184947.057929:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184947.057931:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.057932:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184947.057935:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184947.057937:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104144 00010000:00000001:3.0:1496184947.057939:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184947.057942:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8e050 x1568861104104144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184947.057957:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184947.057958:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184947.057960:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=101 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184947.057963:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184947.057964:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.057966:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.057967:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.057968:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184947.057970:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057971:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184947.057974:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184947.057978:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104144, offset 0 00000100:00000001:3.0:1496184947.057995:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.057996:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.057997:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.057999:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.058000:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184947.058001:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.058002:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184947.058004:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e050 x1568861104104144/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184947.058010:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104144:12345-10.37.248.124@o2ib1:8 Request procesed in 274us (393us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184947.058015:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6090 00000100:00000040:3.0:1496184947.058017:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184947.058019:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184947.058020:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184947.058023:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132fe00. 00000020:00000010:3.0:1496184947.058025:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184947.058027:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88101021d800. 00000020:00000040:3.0:1496184947.058029:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184947.058031:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184947.058034:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184947.058047:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.058048:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184947.069921:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.069923:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.080920:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.080930:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.080931:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.091927:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184947.102663:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184947.102665:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184947.102668:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8f450 x1568861104104176 msgsize 520 00000100:00100000:1.0:1496184947.102674:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184947.102676:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184947.102679:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184947.102713:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.102715:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.102718:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184947.102720:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184947.102721:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184947.102723:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184947.102727:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184947.102730:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104176 00000020:00000001:3.0:1496184947.102732:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184947.102736:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184947.102738:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184947.102741:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:3.0:1496184947.102742:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184947.102744:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184947.102748:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.102749:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184947.102751:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000010:3.0:1496184947.102754:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132fc80. 00000020:00000010:3.0:1496184947.102757:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00000100:00000001:3.0:1496184947.102759:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184947.102760:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184947.102761:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184947.102765:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184947.102771:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8f450 x1568861104104176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.102776:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184947.102778:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102783:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184947.102786:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.102787:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184947.102788:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.102788:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.102790:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184947.102791:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184947.102794:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6091 00000100:00000040:3.0:1496184947.102796:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.102797:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736409680 : -131906973141936 : ffff880803b8f450) 00000100:00000040:3.0:1496184947.102800:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8f450 x1568861104104176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.102805:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.102806:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184947.102808:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104176:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184947.102812:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104176 00000020:00000001:3.0:1496184947.102814:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184947.102817:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184947.102823:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184947.102827:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184947.102829:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184947.102831:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23f9800. 02000000:00000001:3.0:1496184947.102833:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102834:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.102838:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184947.102840:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184947.102842:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184947.102844:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184947.102846:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184947.102848:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:3.0:1496184947.102852:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184947.102854:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184947.102855:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184947.102856:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184947.102858:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184947.102858:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184947.102860:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184947.102861:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184947.102863:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184947.102864:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184947.102865:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184947.102867:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184947.102868:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184947.102872:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184947.102873:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184947.102874:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184947.102878:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184947.102879:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184947.102882:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184947.102883:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184947.102884:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184947.102886:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184947.102888:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184947.102900:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184947.102902:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.102903:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184947.102905:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184947.102907:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184947.102908:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.102909:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.102910:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184947.102912:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184947.102914:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.102915:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184947.102916:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.102918:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.102920:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.102921:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102923:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184947.102926:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184947.102928:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184947.102930:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184947.102932:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.102933:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.102935:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102936:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.102937:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.102939:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.102941:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184947.102943:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184947.102945:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102946:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184947.102948:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184947.102948:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184947.102950:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184947.102951:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184947.102952:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184947.102954:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184947.102955:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.102957:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.102958:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184947.102959:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.102961:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184947.102963:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184947.102965:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184947.102965:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184947.102967:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184947.102968:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184947.102969:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.102971:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184947.102972:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184947.102974:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104176 00010000:00000001:3.0:1496184947.102976:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184947.102978:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8f450 x1568861104104176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184947.102984:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184947.102985:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184947.102987:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=101 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184947.102989:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184947.102991:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.102993:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.102994:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.102996:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184947.102997:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.102999:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184947.103001:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184947.103004:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104176, offset 0 00000100:00000001:3.0:1496184947.103023:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.103024:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.103025:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.103026:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.103028:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184947.103029:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.103030:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184947.103032:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8f450 x1568861104104176/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184947.103037:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104176:12345-10.37.248.124@o2ib1:8 Request procesed in 231us (368us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184947.103042:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6091 00000100:00000040:3.0:1496184947.103044:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184947.103046:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184947.103047:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184947.103049:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132fc80. 00000020:00000010:3.0:1496184947.103051:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184947.103053:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000040:3.0:1496184947.103054:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184947.103056:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184947.103058:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184947.103063:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.103063:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184947.105353:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.105356:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.105375:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.105379:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.105383:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.105388:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.136788:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.136791:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00000001:3.0:1496184947.136811:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.136815:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.136816:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000001:3.0:1496184947.136821:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184947.165776:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.165777:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.165780:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104104240 msgsize 520 00000100:00100000:7.0:1496184947.165785:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.165786:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.165788:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184947.166139:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.166141:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.166142:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77b400. 02000000:00000001:7.0:1496184947.166145:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.166146:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.166605:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.166607:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.166609:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.166614:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.166615:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.166617:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.168558:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.168572:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184947.168578:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.168583:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.168583:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184947.168588:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184947.197079:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.197081:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.197091:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb850 x1568861104104272 msgsize 520 00000100:00100000:7.0:1496184947.197096:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.197097:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.197099:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.199814:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.199816:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184947.210949:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.210954:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.210955:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184947.221969:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.235470:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.235473:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.246535:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.246541:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.246541:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.257589:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184947.257672:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184947.257674:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184947.257677:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e850 x1568861104104336 msgsize 520 00000100:00100000:1.0:1496184947.257684:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184947.257685:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184947.257689:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184947.257695:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.257697:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.257701:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184947.257704:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184947.257706:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184947.257708:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184947.257712:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184947.257716:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104336 00000020:00000001:3.0:1496184947.257719:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184947.257721:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184947.257723:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184947.257726:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184947.257730:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184947.257732:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184947.257736:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.257737:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184947.257740:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000010:3.0:1496184947.257744:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132fa80. 00000020:00000010:3.0:1496184947.257746:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00000100:00000001:3.0:1496184947.257749:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184947.257750:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184947.257752:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184947.257757:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184947.257762:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8e850 x1568861104104336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.257768:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184947.257769:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.257772:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184947.257776:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.257777:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184947.257778:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.257779:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.257780:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184947.257782:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184947.257785:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6092 00000100:00000040:3.0:1496184947.257787:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.257789:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736406608 : -131906973145008 : ffff880803b8e850) 00000100:00000040:3.0:1496184947.257792:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e850 x1568861104104336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.257797:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.257798:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184947.257800:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104104336:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184947.257803:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104336 00000020:00000001:3.0:1496184947.257805:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184947.257809:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184947.257817:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184947.257823:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184947.257824:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184947.257827:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23f8400. 02000000:00000001:3.0:1496184947.257829:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.257831:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.257837:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184947.257839:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184947.257841:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184947.257844:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184947.257846:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184947.257848:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:3.0:1496184947.257852:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184947.257862:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184947.257865:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184947.257867:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184947.257869:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184947.257870:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184947.257872:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184947.257874:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184947.257876:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184947.257877:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184947.257879:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184947.257880:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184947.257882:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184947.257887:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184947.257901:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184947.257903:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184947.257905:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184947.257907:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184947.257909:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184947.257912:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184947.257913:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184947.257914:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184947.257917:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184947.257918:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184947.257920:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.257921:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184947.257923:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184947.257925:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184947.257927:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.257928:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.257929:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184947.257932:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184947.257934:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.257935:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184947.257936:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.257938:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.257940:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.257941:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.257943:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184947.257947:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184947.257949:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184947.257952:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184947.257954:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.257955:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.257957:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.257958:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.257959:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.257960:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.257962:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184947.257965:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184947.257966:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.257969:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184947.257971:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184947.257973:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184947.257974:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184947.257976:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184947.257977:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184947.257979:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184947.257981:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.257983:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.257984:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184947.257986:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.257989:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184947.257991:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184947.257992:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184947.257993:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184947.257995:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184947.257996:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184947.257998:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.257999:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184947.258002:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184947.258003:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104336 00010000:00000001:3.0:1496184947.258006:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184947.258008:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8e850 x1568861104104336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184947.258021:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184947.258022:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184947.258023:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=101 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184947.258027:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184947.258029:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.258030:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.258031:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.258032:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184947.258034:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.258035:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184947.258038:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184947.258042:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104336, offset 0 00000100:00000001:3.0:1496184947.258059:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.258060:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.258061:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.258062:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.258063:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184947.258065:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.258066:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184947.258068:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e850 x1568861104104336/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184947.258073:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104336:12345-10.37.248.124@o2ib1:8 Request procesed in 275us (395us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184947.258079:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6092 00000100:00000040:3.0:1496184947.258081:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184947.258082:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184947.258083:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184947.258086:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132fa80. 00000020:00000010:3.0:1496184947.258089:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184947.258091:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000040:3.0:1496184947.258099:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184947.258101:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184947.258104:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184947.258109:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.258111:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.258446:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.258450:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.258464:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23f8400. 02000000:00000001:7.0:1496184947.258473:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.258474:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.258851:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.258853:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.258855:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026078450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.258860:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.258862:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.258864:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.270160:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.270162:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:7.0:1496184947.277954:0:13694:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:7.0:1496184947.277956:0:13694:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:7.0:1496184947.277960:0:13694:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:7.0:1496184947.277961:0:13694:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184947.277962:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184947.277964:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184947.277965:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184947.277966:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184947.281005:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184947.281007:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184947.281010:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184947.281010:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184947.281017:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184947.281019:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184947.281021:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184947.281021:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.281159:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.281164:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.281165:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.292145:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184947.303047:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.303049:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.303051:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc050 x1568861104104368 msgsize 520 00000100:00100000:7.0:1496184947.303057:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.303058:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.303060:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184947.303407:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.303410:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.303411:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77f800. 02000000:00000001:7.0:1496184947.303414:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.303415:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.303798:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.303800:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.303802:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.303807:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.303808:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.303811:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.305645:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.305648:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.305667:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.305671:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.305674:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.305680:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.336400:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.336403:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184947.336412:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.336429:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.336430:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184947.336435:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:2.0:1496184947.341929:0:13691:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1496184947.341932:0:13691:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184947.341934:0:13689:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184947.341937:0:13689:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184947.341942:0:13691:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184947.341944:0:13691:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.341944:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184947.341945:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184947.341947:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184947.341947:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184947.341951:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.341951:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184947.341953:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184947.341953:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184947.341955:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.341955:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184947.342002:0:13690:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1496184947.342004:0:13690:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184947.342008:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1496184947.342009:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184947.342010:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184947.342012:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184947.342013:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184947.342014:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.350944:0:13692:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184947.350948:0:13692:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184947.350953:0:13692:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184947.350954:0:13692:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.350955:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184947.350957:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.350958:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184947.350959:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.367260:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.367263:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.367282:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.367286:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.367287:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.367292:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:2.0:1496184947.389694:0:6131:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:2.0:1496184947.389696:0:6131:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000040:2.0:1496184947.389699:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8807f6ee7850 x1568861152397696 msgsize 224 00000100:00100000:2.0:1496184947.389706:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184947.389707:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:2.0:1496184947.389710:0:6131:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:1.0:1496184947.389743:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184947.389746:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184947.389750:0:14306:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:1.0:1496184947.389752:0:14306:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:1.0:1496184947.389754:0:14306:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:1.0:1496184947.389756:0:14306:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1496184947.389760:0:14306:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1496184947.389763:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397696 00000020:00000001:1.0:1496184947.389766:0:14306:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:1.0:1496184947.389767:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf61f 00000020:00000001:1.0:1496184947.389770:0:14306:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:1.0:1496184947.389772:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff8807f23fa400 : new refcount 5 00000020:00000001:1.0:1496184947.389774:0:14306:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612166443246592 : -131907266305024 : ffff8807f23fa400) 00000020:00000001:1.0:1496184947.389775:0:14306:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612166443246592 : -131907266305024 : ffff8807f23fa400) 00000100:00000001:1.0:1496184947.389780:0:14306:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:1.0:1496184947.389781:0:14306:0:(service.c:1128:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:1.0:1496184947.389784:0:14306:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100fcbd200. 00000020:00000010:1.0:1496184947.389787:0:14306:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100e364b80. 00000020:00000010:1.0:1496184947.389789:0:14306:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88100eeb7e40. 00000100:00000040:1.0:1496184947.389792:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00000001:1.0:1496184947.389794:0:14306:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:1.0:1496184947.389795:0:14306:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:1.0:1496184947.389796:0:14306:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389801:0:14306:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389806:0:14306:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:1.0:1496184947.389810:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184947.389811:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:1.0:1496184947.389812:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184947.389813:0:13733:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184947.389813:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184947.389814:0:14306:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00004000:00000001:2.0:1496184947.389815:0:13733:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184947.389817:0:13733:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184947.389818:0:13733:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:1.0:1496184947.389818:0:14306:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:1.0:1496184947.389821:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 959 00000100:00000001:2.0:1496184947.389823:0:13733:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:1.0:1496184947.389823:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8807f23fa400 : new rpc_count 1 00000100:00000001:1.0:1496184947.389824:0:14306:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166521813072 : -131907187738544 : ffff8807f6ee7850) 00000100:00000001:2.0:1496184947.389825:0:13733:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:2.0:1496184947.389827:0:13733:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184947.389828:0:13733:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000040:1.0:1496184947.389829:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8807f6ee7850 x1568861152397696/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:698/0 lens 224/0 e 0 to 0 dl 1496184953 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00000001:1.0:1496184947.389835:0:14306:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:1.0:1496184947.389836:0:14306:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:1.0:1496184947.389838:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397696:12345-10.37.248.197@o2ib1:13 00000100:00000200:1.0:1496184947.389841:0:14306:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861152397696 00000020:00000001:1.0:1496184947.389843:0:14306:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:1.0:1496184947.389846:0:14306:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:1.0:1496184947.389847:0:14306:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184947.389849:0:14306:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:1.0:1496184947.389850:0:14306:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284592 : -1589267024 : ffffffffa145b5b0) 00000020:00000001:1.0:1496184947.389852:0:14306:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:1.0:1496184947.389854:0:14306:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1496184947.389855:0:14306:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:1.0:1496184947.389856:0:14306:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:1.0:1496184947.389858:0:14306:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389861:0:14306:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:1.0:1496184947.389863:0:14306:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:1.0:1496184947.389865:0:14306:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 624 at ffff880174bcf400. 02000000:00000001:1.0:1496184947.389867:0:14306:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389868:0:14306:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:1.0:1496184947.389870:0:14306:0:(ofd_dev.c:1916:ofd_statfs_hdl()) Process entered 00002000:00000001:1.0:1496184947.389873:0:14306:0:(ofd_obd.c:725:ofd_statfs()) Process entered 00002000:00000001:1.0:1496184947.389875:0:14306:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:1.0:1496184947.389876:0:14306:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000024:1.0:1496184947.389877:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 9758656 pending 0 free 80206888960 avail 76094111744 00002000:00000020:1.0:1496184947.389880:0:14306:0:(tgt_grant.c:150:tgt_check_export_grants()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 dirty 0 pend 0 grant 9510912 00002000:00000020:1.0:1496184947.389883:0:14306:0:(tgt_grant.c:150:tgt_check_export_grants()) lustre-OST0001: cli lustre-MDT0000-mdtlov_UUID/ffff8807f23fa400 dirty 0 pend 0 grant 0 00002000:00000020:1.0:1496184947.389885:0:14306:0:(tgt_grant.c:144:tgt_check_export_grants()) lustre-OST0001: processing self export: 247744 0 0 00002000:00000020:1.0:1496184947.389888:0:14306:0:(tgt_grant.c:150:tgt_check_export_grants()) lustre-OST0001: cli lustre-OST0001_UUID/ffff8810371fe800 dirty 0 pend 0 grant 247744 00002000:00000020:1.0:1496184947.389900:0:14306:0:(ofd_obd.c:760:ofd_statfs()) 19595366 blocks: 19581760 free, 18577604 avail; 1173120 objects: 1172809 free; state 0 00002000:00000001:1.0:1496184947.389902:0:14306:0:(ofd_obd.c:794:ofd_statfs()) Process leaving 00002000:00000001:1.0:1496184947.389904:0:14306:0:(ofd_dev.c:1932:ofd_statfs_hdl()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:1.0:1496184947.389906:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968339, transno 0, xid 1568861152397696 00010000:00000001:1.0:1496184947.389908:0:14306:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:1.0:1496184947.389911:0:14306:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8807f6ee7850 x1568861152397696/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:698/0 lens 224/368 e 0 to 0 dl 1496184953 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00000001:1.0:1496184947.389917:0:14306:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:1.0:1496184947.389919:0:14306:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:1.0:1496184947.389921:0:14306:0:(import.c:1678:at_measured()) add 1 to ffff8807f6ee7d50 time=101 v=1 (1 1 1 1) 00000100:00000001:1.0:1496184947.389924:0:14306:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:1.0:1496184947.389926:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000001:1.0:1496184947.389927:0:14306:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612202089699200 : -131871619852416 : ffff88103ef15780) 02000000:00000001:1.0:1496184947.389929:0:14306:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:1.0:1496184947.389930:0:14306:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389932:0:14306:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:1.0:1496184947.389934:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000200:1.0:1496184947.389937:0:14306:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1568861152397696, offset 192 00000100:00000001:1.0:1496184947.389953:0:14306:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184947.389954:0:14306:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:1.0:1496184947.389956:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000001:1.0:1496184947.389957:0:14306:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:1.0:1496184947.389958:0:14306:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:1.0:1496184947.389959:0:14306:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184947.389960:0:14306:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:1.0:1496184947.389963:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8807f6ee7850 x1568861152397696/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:698/0 lens 224/336 e 0 to 0 dl 1496184953 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000001:2.0:1496184947.389964:0:6132:0:(events.c:394:reply_out_callback()) Process entered 00000100:00000001:2.0:1496184947.389965:0:6132:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00100000:1.0:1496184947.389968:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13023:x1568861152397696:12345-10.37.248.197@o2ib1:13 Request procesed in 132us (268us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184947.389973:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 959 00000100:00000040:1.0:1496184947.389975:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8807f23fa400 : new rpc_count 0 00000100:00000001:1.0:1496184947.389977:0:14306:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:1.0:1496184947.389979:0:14306:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:1.0:1496184947.389982:0:14306:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100e364b80. 00000020:00000010:1.0:1496184947.389984:0:14306:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88100eeb7e40. 00000020:00000010:1.0:1496184947.389986:0:14306:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100fcbd200. 00000020:00000040:1.0:1496184947.389988:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff8807f23fa400 : new refcount 4 02000000:00000001:1.0:1496184947.389990:0:14306:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:1.0:1496184947.389992:0:14306:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 624 at ffff880174bcf400. 02000000:00000001:1.0:1496184947.389993:0:14306:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:1.0:1496184947.389994:0:14306:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:1.0:1496184947.389996:0:14306:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00004000:00000001:1.0:1496184947.390001:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184947.390002:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.394609:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.394611:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.394613:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdcc50 x1568861104104464 msgsize 520 00000100:00100000:7.0:1496184947.394619:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.394620:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.394622:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.397302:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.397305:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184947.408454:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.408459:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.408460:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184947.419463:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.430520:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.430522:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.441561:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.441566:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.441567:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.452598:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:2.0:1496184947.452682:0:109:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:2.0:1496184947.452685:0:109:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:2.0:1496184947.452688:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d050 x1568861104104528 msgsize 520 00000100:00100000:2.0:1496184947.452694:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184947.452695:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:2.0:1496184947.452699:0:109:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184947.452705:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.452708:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.452712:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184947.452714:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184947.452716:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184947.452718:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184947.452722:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184947.452726:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104528 00000020:00000001:3.0:1496184947.452729:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184947.452730:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184947.452733:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184947.452735:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184947.452740:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184947.452742:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184947.452747:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.452748:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184947.452750:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000010:3.0:1496184947.452754:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132f900. 00000020:00000010:3.0:1496184947.452757:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00000100:00000001:3.0:1496184947.452759:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184947.452760:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184947.452762:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184947.452767:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184947.452772:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8d050 x1568861104104528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.452778:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184947.452779:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.452784:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184947.452787:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.452788:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184947.452790:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184947.452790:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184947.452792:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184947.452793:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184947.452797:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6093 00000100:00000040:3.0:1496184947.452799:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.452801:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736400464 : -131906973151152 : ffff880803b8d050) 00000100:00000040:3.0:1496184947.452803:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d050 x1568861104104528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184947.452809:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184947.452810:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184947.452812:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104528:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184947.452815:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104528 00000020:00000001:3.0:1496184947.452817:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184947.452821:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184947.452829:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184947.452835:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184947.452836:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184947.452839:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fe000. 02000000:00000001:3.0:1496184947.452841:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.452842:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.452855:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184947.452858:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184947.452861:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184947.452863:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184947.452865:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184947.452867:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:3.0:1496184947.452871:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184947.452874:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184947.452877:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184947.452879:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184947.452881:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184947.452882:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184947.452884:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184947.452886:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184947.452888:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184947.452896:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184947.452898:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184947.452900:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184947.452901:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184947.452906:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184947.452908:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184947.452909:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184947.452912:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184947.452913:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184947.452916:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184947.452919:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184947.452920:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184947.452921:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184947.452923:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184947.452925:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184947.452926:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.452928:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184947.452929:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184947.452931:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184947.452933:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184947.452934:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.452935:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184947.452937:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184947.452939:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.452941:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184947.452942:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184947.452944:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.452946:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.452947:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.452949:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184947.452953:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184947.452955:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184947.452958:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184947.452960:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.452961:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.452963:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.452964:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.452965:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.452966:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.452968:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184947.452971:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184947.452973:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.452975:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184947.452978:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184947.452979:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184947.452981:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184947.452983:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184947.452984:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184947.452985:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184947.452987:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184947.452989:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184947.452990:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184947.452992:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.452995:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184947.452997:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184947.452999:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184947.453000:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184947.453001:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184947.453003:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184947.453004:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.453006:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184947.453009:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184947.453011:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104528 00010000:00000001:3.0:1496184947.453013:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184947.453016:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8d050 x1568861104104528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184947.453022:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184947.453024:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184947.453025:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=101 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184947.453028:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184947.453030:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184947.453032:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.453033:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184947.453043:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184947.453044:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.453046:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184947.453048:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184947.453052:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104528, offset 0 00000100:00000001:3.0:1496184947.453069:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.453070:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184947.453072:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184947.453073:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184947.453074:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184947.453075:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184947.453076:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184947.453078:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d050 x1568861104104528/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184947.453084:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104528:12345-10.37.248.124@o2ib1:8 Request procesed in 273us (394us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184947.453089:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6093 00000100:00000040:3.0:1496184947.453091:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184947.453092:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184947.453093:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184947.453096:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132f900. 00000020:00000010:3.0:1496184947.453098:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184947.453100:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010a19000. 00000020:00000040:3.0:1496184947.453101:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184947.453103:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184947.453106:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184947.453111:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184947.453112:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184947.455796:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.455799:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.466798:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.466802:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.466803:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.477773:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184947.493234:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.493236:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.493237:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e778800. 02000000:00000001:7.0:1496184947.493240:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.493241:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.493686:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.493688:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.493690:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.493695:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.493696:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.493706:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184947.495564:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.495567:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.506594:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.506599:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.506603:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ca00 00000100:00000001:3.0:1496184947.517586:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184947.523730:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.523732:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.523734:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd850 x1568861104104592 msgsize 520 00000100:00100000:7.0:1496184947.523739:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.523740:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.523742:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184947.524149:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.524152:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.524165:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77c400. 02000000:00000001:7.0:1496184947.524168:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.524169:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.524597:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.524599:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.524601:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.524613:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.524614:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.524616:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000020:00000001:6.0:1496184947.524834:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.524835:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.524837:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.524838:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.524839:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.524840:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c800. 00000100:00000010:6.0:1496184947.524841:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.524843:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.524847:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.524848:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.524874:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe4a0-0x592def0cfe4a0 00000100:00000001:6.0:1496184947.524877:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.526465:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.526468:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.537509:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.537514:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.537515:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184947.548507:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.548575:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.548593:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.548595:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.548599:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.548600:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.548602:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.548603:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.548637:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.548639:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.548641:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.548642:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.548642:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.548644:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.548645:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.548646:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.548647:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.548647:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.548649:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.548650:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.548652:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c800. 00000100:00000001:6.0:1496184947.548653:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.548654:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.548655:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779000. 02000000:00000001:6.0:1496184947.548657:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.548659:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.548660:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.548663:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.548668:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.548669:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.548670:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.548671:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.548672:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.548674:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.548680:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23930us (24077us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.548685:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57615 00000100:00000040:6.0:1496184947.548686:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.548688:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.548689:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.548691:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.548697:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.548697:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.548699:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.548700:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.548701:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.548703:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.548704:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.548705:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.548705:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.548706:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00002000:00000001:6.0:1496184947.548707:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.548709:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.548710:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808226bcc00. 00000020:00000010:6.0:1496184947.548712:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.548713:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.548715:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.548716:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.548718:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.548724:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.548725:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.554594:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.554599:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.554601:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.554602:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.554603:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.554604:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.554606:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.554608:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104624 00000020:00000001:6.0:1496184947.554610:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.554611:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.554612:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.554613:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.554615:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.554616:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.554618:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.554619:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.554621:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.554623:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808226bcb80. 00000020:00000010:6.0:1496184947.554624:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.554626:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.554627:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.554628:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.554630:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.554633:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bddc50 x1568861104104624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.554642:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.554643:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554647:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.554649:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.554650:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.554651:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.554652:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.554653:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.554653:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.554656:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43361 00000100:00000040:6.0:1496184947.554657:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.554659:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683672144 : -131872025879472 : ffff881026bddc50) 00000100:00000040:6.0:1496184947.554661:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104104624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.554666:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.554667:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.554669:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104624:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.554671:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104624 00000020:00000001:6.0:1496184947.554677:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.554679:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.554682:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.554684:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.554685:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.554687:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779000. 02000000:00000001:6.0:1496184947.554688:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554689:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.554691:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.554693:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.554695:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.554696:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.554698:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.554700:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.554702:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.554704:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.554705:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.554706:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.554707:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.554707:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.554709:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.554710:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.554711:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.554712:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.554713:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.554715:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.554716:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.554719:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.554720:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.554721:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.554723:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.554724:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.554726:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.554727:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.554728:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.554729:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.554736:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.554737:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.554739:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.554740:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.554741:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.554742:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.554744:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.554745:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.554746:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.554747:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.554749:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.554750:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.554751:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.554752:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.554753:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.554755:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554756:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.554758:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.554760:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.554762:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.554763:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.554764:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.554765:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554767:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.554768:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.554769:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.554771:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.554772:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.554773:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554774:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.554775:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.554776:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.554777:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.554779:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.554779:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.554780:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.554782:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.554783:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.554784:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.554786:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.554788:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.554789:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.554790:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.554791:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.554792:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.554793:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.554794:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.554795:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.554796:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.554797:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104624 00010000:00000001:6.0:1496184947.554799:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.554800:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bddc50 x1568861104104624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.554806:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.554806:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.554808:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.554810:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.554811:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.554813:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.554814:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.554815:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.554816:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554817:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.554819:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.554821:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104624, offset 0 00000100:00000001:6.0:1496184947.554832:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.554833:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.554834:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.554835:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.554836:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.554837:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.554838:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.554840:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104104624/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.554845:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104624:12345-10.37.248.124@o2ib1:8 Request procesed in 178us (289us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.554850:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43361 00000100:00000040:6.0:1496184947.554851:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.554853:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.554853:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.554855:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808226bcb80. 00000020:00000010:6.0:1496184947.554856:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.554858:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.554859:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.554861:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.554862:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.554866:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.554866:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.554932:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.554934:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.554936:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779000. 02000000:00000001:7.0:1496184947.554939:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.554940:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.555427:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.555429:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.555431:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.555436:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.555442:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.555444:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.555476:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.555478:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.555480:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.555481:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.555482:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.555483:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.555485:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.555487:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.555489:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.555489:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.555491:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.555492:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.555493:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.555495:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.555497:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.555498:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.555499:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.555502:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808226bca00. 00000020:00000010:6.0:1496184947.555503:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.555505:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.555507:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.555507:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.555509:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.555510:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.555511:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.555513:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.555514:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.555515:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.555516:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.555517:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.555518:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.555519:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.555520:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.555521:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.555521:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555522:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.555524:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555525:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555526:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.555527:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.555528:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.555529:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.555532:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.555538:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.555539:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.555541:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.555542:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.555542:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.555544:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.555546:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.555546:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.555547:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.555547:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00002000:00000001:6.0:1496184947.555549:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.555550:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.555552:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.555557:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.555559:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.555560:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.555561:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.555562:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.555562:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.555563:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.555566:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57616 00000100:00000040:6.0:1496184947.555567:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.555569:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671710800 : -131872037840816 : ffff881026075850) 00000100:00000040:6.0:1496184947.555571:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.555576:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.555577:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.555579:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.555581:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.555582:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.555584:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.555585:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555586:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.555586:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.555588:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.555589:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.555590:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.555591:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.555591:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555592:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.555594:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.555595:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.555596:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77c800. 02000000:00000001:6.0:1496184947.555597:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.555599:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555600:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.555601:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.555603:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.555607:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.555610:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.555610:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.555613:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.555615:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.555616:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.555616:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.555618:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.555620:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.555620:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.555623:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.555624:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.555625:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.555625:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.555626:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.555628:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.555629:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.555630:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.555632:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.555654:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.555660:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555661:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.555664:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.555665:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.555666:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.555667:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.555668:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.555669:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7fc00. 00000100:00000010:6.0:1496184947.555671:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.555672:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.555677:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.555678:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.555704:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe4c0-0x592def0cfe4c0 00000100:00000001:6.0:1496184947.555706:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.557404:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.557406:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fc00 00000100:00000001:3.0:1496184947.568421:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.568426:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.568426:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fc00 00000100:00000001:3.0:1496184947.579411:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.579473:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.579493:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.579495:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.579499:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.579500:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.579502:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.579503:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.579536:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.579539:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.579541:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.579541:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.579542:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.579544:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.579545:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.579546:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.579547:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.579547:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.579549:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.579550:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.579552:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7fc00. 00000100:00000001:6.0:1496184947.579553:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.579554:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.579555:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77c800. 02000000:00000001:6.0:1496184947.579556:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.579558:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.579560:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.579562:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.579567:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.579569:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.579570:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.579571:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.579572:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.579574:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.579580:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24002us (24147us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.579585:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57616 00000100:00000040:6.0:1496184947.579587:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.579588:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.579589:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.579592:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.579597:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.579598:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.579599:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.579600:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.579601:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.579603:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.579605:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.579605:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.579606:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.579606:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00002000:00000001:6.0:1496184947.579608:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.579609:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.579610:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808226bca00. 00000020:00000010:6.0:1496184947.579612:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.579613:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.579615:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.579616:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.579619:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.579624:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.579625:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.590110:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.590112:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.590114:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde450 x1568861104104656 msgsize 520 00000100:00100000:7.0:1496184947.590119:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.590120:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.590123:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.590155:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.590158:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.590159:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.590161:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.590162:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.590163:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.590165:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.590167:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104656 00000020:00000001:6.0:1496184947.590169:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.590169:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.590171:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.590172:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.590173:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.590175:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.590177:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.590178:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.590179:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.590183:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232200. 00000020:00000010:6.0:1496184947.590185:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.590187:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.590187:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.590188:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.590191:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.590194:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde450 x1568861104104656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.590199:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.590200:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590204:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.590206:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.590207:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.590208:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.590209:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.590210:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.590211:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.590213:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43362 00000100:00000040:6.0:1496184947.590215:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.590216:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683674192 : -131872025877424 : ffff881026bde450) 00000100:00000040:6.0:1496184947.590218:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104104656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.590223:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.590224:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.590226:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104656:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.590228:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104656 00000020:00000001:6.0:1496184947.590229:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.590231:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.590234:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.590237:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.590238:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.590239:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77c800. 02000000:00000001:6.0:1496184947.590240:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590241:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.590244:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.590245:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.590247:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.590248:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.590250:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.590252:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.590255:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.590256:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.590257:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.590258:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.590259:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.590259:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.590261:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.590262:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.590263:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.590264:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.590265:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.590266:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.590268:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.590271:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.590272:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.590273:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.590275:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.590276:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.590278:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.590279:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.590280:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.590281:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.590283:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.590284:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.590286:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.590287:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.590288:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.590289:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.590291:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.590292:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.590293:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.590294:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.590295:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.590296:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.590297:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.590298:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.590300:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.590301:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590302:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.590305:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.590308:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.590309:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.590311:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.590312:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.590313:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590314:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.590315:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.590316:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.590318:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.590319:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.590320:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590321:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.590323:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.590323:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.590324:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.590326:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.590326:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.590328:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.590329:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.590331:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.590332:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.590333:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.590335:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.590336:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.590337:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.590338:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.590339:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.590340:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.590340:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.590342:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.590343:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.590344:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104656 00010000:00000001:6.0:1496184947.590345:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.590348:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde450 x1568861104104656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.590353:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.590354:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.590355:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.590357:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.590359:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.590360:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.590361:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.590363:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.590363:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590365:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.590366:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.590368:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104656, offset 0 00000100:00000001:6.0:1496184947.590379:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.590380:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.590381:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.590383:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.590384:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.590384:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.590385:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.590387:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104104656/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.590393:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104656:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.590397:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43362 00000100:00000040:6.0:1496184947.590399:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.590400:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.590401:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.590402:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232200. 00000020:00000010:6.0:1496184947.590404:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.590405:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.590407:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.590408:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.590410:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.590413:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.590414:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.590484:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.590487:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.590490:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77c800. 02000000:00000001:7.0:1496184947.590494:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.590495:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.590995:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.590997:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.591007:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.591012:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.591013:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.591016:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.591048:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.591051:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.591052:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.591053:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.591054:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.591055:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.591058:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.591060:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.591061:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.591062:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.591063:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.591064:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.591066:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.591068:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.591070:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.591071:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.591072:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.591074:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232580. 00000020:00000010:6.0:1496184947.591076:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.591078:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.591079:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.591080:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.591081:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.591082:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.591083:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.591085:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.591086:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.591087:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.591088:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.591089:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.591090:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.591091:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.591092:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.591093:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.591094:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591095:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.591096:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591097:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591098:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.591099:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.591100:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.591101:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.591104:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.591111:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.591111:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.591113:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.591114:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.591115:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.591116:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.591118:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.591118:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.591119:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.591119:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00002000:00000001:6.0:1496184947.591121:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.591122:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.591124:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.591129:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.591131:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.591131:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.591132:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.591133:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.591134:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.591135:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.591137:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57617 00000100:00000040:6.0:1496184947.591139:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.591140:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671703632 : -131872037847984 : ffff881026073c50) 00000100:00000040:6.0:1496184947.591143:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.591148:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.591149:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.591150:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.591152:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.591153:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.591155:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.591156:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591157:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.591158:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.591159:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.591160:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.591161:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.591162:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.591162:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591164:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.591165:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.591166:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.591168:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77b800. 02000000:00000001:6.0:1496184947.591169:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.591170:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591171:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.591172:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.591174:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.591178:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.591180:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.591181:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.591183:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.591185:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.591186:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.591187:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.591188:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.591190:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.591191:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.591193:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.591195:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.591195:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.591196:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.591197:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.591199:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.591199:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.591200:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.591203:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.591223:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.591228:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591230:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.591232:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.591234:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.591235:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.591236:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.591237:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.591238:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d600. 00000100:00000010:6.0:1496184947.591240:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.591241:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.591245:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.591246:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.591273:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe4e0-0x592def0cfe4e0 00000100:00000001:6.0:1496184947.591275:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.592963:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.592966:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184947.603997:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.604002:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.604002:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000001:2.0:1496184947.613928:0:13688:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1496184947.613931:0:13688:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184947.613939:0:13688:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184947.613940:0:13688:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184947.613946:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184947.613949:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184947.613951:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184947.613952:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.614976:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.615048:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.615054:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.615056:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.615060:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.615061:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.615063:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.615065:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.615099:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.615101:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.615103:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.615104:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.615104:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.615106:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.615107:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.615108:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.615109:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.615109:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.615111:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.615112:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.615114:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d600. 00000100:00000001:6.0:1496184947.615115:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.615116:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.615117:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77b800. 02000000:00000001:6.0:1496184947.615118:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.615120:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.615122:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.615124:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.615129:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.615131:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.615132:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.615132:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.615133:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.615135:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.615141:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23992us (24132us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.615146:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57617 00000100:00000040:6.0:1496184947.615148:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.615149:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.615150:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.615153:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.615159:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.615159:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.615161:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.615162:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.615163:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.615165:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.615166:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.615167:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.615167:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.615168:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00002000:00000001:6.0:1496184947.615169:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.615170:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.615172:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232580. 00000020:00000010:6.0:1496184947.615174:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.615175:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.615177:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.615178:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.615180:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.615186:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.615187:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.625951:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.625953:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.625955:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde050 x1568861104104688 msgsize 520 00000100:00100000:7.0:1496184947.625960:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.625961:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.625964:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.625997:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.625999:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.626001:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.626002:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.626003:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.626004:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.626006:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.626008:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104688 00000020:00000001:6.0:1496184947.626010:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.626011:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.626012:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.626013:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.626015:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.626016:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.626018:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.626019:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.626021:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.626023:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232b00. 00000020:00000010:6.0:1496184947.626024:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.626026:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.626027:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.626027:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.626030:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.626033:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde050 x1568861104104688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.626038:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.626039:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626043:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.626045:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.626046:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.626047:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.626048:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.626049:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.626050:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.626052:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43363 00000100:00000040:6.0:1496184947.626054:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.626055:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683673168 : -131872025878448 : ffff881026bde050) 00000100:00000040:6.0:1496184947.626057:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104104688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.626062:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.626063:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.626065:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104688:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.626068:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104688 00000020:00000001:6.0:1496184947.626070:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.626072:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.626075:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.626077:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.626078:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.626080:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77b800. 02000000:00000001:6.0:1496184947.626081:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626082:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.626084:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.626086:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.626088:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.626089:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.626091:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.626093:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.626095:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.626096:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.626097:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.626098:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.626099:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.626100:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.626101:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.626103:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.626104:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.626105:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.626106:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.626107:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.626108:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.626112:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.626112:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.626114:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.626117:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.626117:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.626120:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.626121:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.626122:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.626123:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.626125:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.626126:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.626127:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.626128:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.626129:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.626131:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.626132:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.626133:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.626134:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.626136:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.626137:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.626138:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.626139:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.626140:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.626142:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.626143:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626144:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.626147:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.626149:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.626150:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.626152:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.626153:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.626154:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626155:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.626157:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.626158:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.626160:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.626161:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.626162:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626163:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.626164:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.626165:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.626166:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.626168:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.626168:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.626169:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.626171:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.626172:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.626173:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.626174:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626176:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.626177:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.626179:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.626179:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.626180:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.626181:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.626182:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626183:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.626184:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.626186:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104688 00010000:00000001:6.0:1496184947.626187:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.626189:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde050 x1568861104104688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.626195:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.626195:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.626197:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.626199:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.626200:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.626202:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.626203:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.626204:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.626205:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626206:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.626208:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.626210:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104688, offset 0 00000100:00000001:6.0:1496184947.626220:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626222:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.626223:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.626224:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.626225:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.626226:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626227:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.626228:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104104688/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.626234:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104688:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.626238:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43363 00000100:00000040:6.0:1496184947.626240:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.626241:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.626242:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.626244:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232b00. 00000020:00000010:6.0:1496184947.626245:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.626247:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.626248:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.626249:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.626251:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.626255:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.626255:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.626320:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.626323:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.626325:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77b800. 02000000:00000001:7.0:1496184947.626329:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.626330:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.626838:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.626847:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.626849:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.626854:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.626855:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.626857:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.626887:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.626903:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.626905:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.626906:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.626907:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.626908:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.626911:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.626912:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.626914:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.626915:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.626916:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.626917:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.626919:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.626920:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.626922:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.626923:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.626925:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.626927:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232300. 00000020:00000010:6.0:1496184947.626928:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.626930:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.626932:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.626933:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.626934:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.626935:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.626936:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.626937:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.626938:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.626939:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.626940:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.626942:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.626942:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.626944:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.626944:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.626945:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.626946:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626947:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.626948:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626949:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.626950:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.626951:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.626952:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.626954:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.626956:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.626963:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.626963:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.626965:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.626966:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.626967:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.626968:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.626970:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.626971:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.626971:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.626972:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00002000:00000001:6.0:1496184947.626973:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626974:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626976:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.626981:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.626983:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.626983:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.626984:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.626985:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.626986:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.626987:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.626990:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57618 00000100:00000040:6.0:1496184947.626991:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.626992:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671717968 : -131872037833648 : ffff881026077450) 00000100:00000040:6.0:1496184947.626995:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.627000:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.627001:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.627003:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.627005:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.627006:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.627007:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.627008:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.627009:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.627010:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.627011:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.627012:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.627013:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.627014:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.627015:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.627016:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.627018:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.627018:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.627020:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e779400. 02000000:00000001:6.0:1496184947.627021:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.627022:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.627024:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.627024:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.627026:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.627030:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.627033:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.627034:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.627036:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.627038:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.627039:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.627040:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.627041:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.627043:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.627044:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.627046:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.627047:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.627048:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.627048:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.627049:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.627051:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.627052:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.627053:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.627055:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.627077:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.627083:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.627084:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.627087:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.627088:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.627089:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.627090:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.627091:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.627092:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7ec00. 00000100:00000010:6.0:1496184947.627094:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.627095:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.627100:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.627101:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.627128:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe500-0x592def0cfe500 00000100:00000001:6.0:1496184947.627130:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.628732:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.628734:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184947.639738:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.639743:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.639744:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184947.650725:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.650785:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.650804:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.650806:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.650810:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.650811:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.650813:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.650815:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.650848:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.650850:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.650852:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.650853:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.650854:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.650855:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.650856:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.650858:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.650859:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.650859:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.650861:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.650862:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.650864:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7ec00. 00000100:00000001:6.0:1496184947.650865:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.650866:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.650867:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779400. 02000000:00000001:6.0:1496184947.650868:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.650870:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.650872:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.650875:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.650880:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.650881:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.650882:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.650883:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.650884:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.650886:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.650902:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23901us (24051us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.650907:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57618 00000100:00000040:6.0:1496184947.650909:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.650911:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.650912:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.650915:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.650920:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.650921:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.650923:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.650924:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.650925:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.650926:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.650928:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.650928:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.650929:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.650930:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00002000:00000001:6.0:1496184947.650931:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.650932:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.650934:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232300. 00000020:00000010:6.0:1496184947.650935:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.650937:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.650938:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.650940:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.650942:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.650947:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.650948:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.661828:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.661830:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.661832:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde850 x1568861104104720 msgsize 520 00000100:00100000:7.0:1496184947.661837:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.661839:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.661841:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.661876:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.661879:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.661880:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.661881:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.661882:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.661883:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.661886:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.661888:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104720 00000020:00000001:6.0:1496184947.661900:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.661901:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.661902:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.661903:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.661905:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.661906:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.661909:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.661909:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.661911:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.661913:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232100. 00000020:00000010:6.0:1496184947.661914:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.661916:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.661917:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.661918:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.661920:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.661923:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde850 x1568861104104720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.661928:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.661929:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.661933:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.661935:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.661936:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.661937:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.661937:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.661938:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.661939:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.661941:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43364 00000100:00000040:6.0:1496184947.661943:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.661944:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683675216 : -131872025876400 : ffff881026bde850) 00000100:00000040:6.0:1496184947.661946:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104104720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.661952:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.661952:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.661954:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104720:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.661956:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104720 00000020:00000001:6.0:1496184947.661957:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.661959:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.661962:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.661965:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.661966:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.661967:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779400. 02000000:00000001:6.0:1496184947.661968:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.661969:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.661972:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.661974:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.661976:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.661977:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.661978:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.661981:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.661983:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.661984:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.661985:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.661986:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.661987:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.661988:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.661989:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.661991:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.661992:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.661993:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.661994:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.661995:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.661996:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.661999:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.662000:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.662002:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.662003:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.662004:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.662006:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.662008:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.662008:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.662009:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.662011:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.662013:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.662014:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662015:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.662016:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.662018:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.662019:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662020:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.662021:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.662023:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.662024:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.662025:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.662026:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.662027:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.662029:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.662030:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662031:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.662034:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.662036:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.662037:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.662039:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.662040:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.662041:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662042:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.662043:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.662045:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.662046:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.662047:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.662048:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662050:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.662051:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.662051:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.662052:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.662054:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.662055:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.662056:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.662058:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.662059:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.662060:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.662061:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662063:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.662064:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.662066:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.662066:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.662067:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.662068:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.662069:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662070:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.662071:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.662072:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104720 00010000:00000001:6.0:1496184947.662074:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.662076:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde850 x1568861104104720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.662081:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.662082:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.662083:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.662085:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.662087:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.662088:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.662089:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.662091:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.662091:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662092:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.662094:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.662096:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104720, offset 0 00000100:00000001:6.0:1496184947.662107:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662108:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.662109:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.662110:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.662111:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.662112:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662113:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.662115:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104104720/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.662120:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104720:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (286us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.662125:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43364 00000100:00000040:6.0:1496184947.662126:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.662128:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.662129:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.662130:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232100. 00000020:00000010:6.0:1496184947.662132:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.662133:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.662135:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.662136:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.662138:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.662141:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.662142:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.662196:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.662198:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.662199:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779400. 02000000:00000001:7.0:1496184947.662202:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.662203:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.662713:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.662715:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.662717:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.662722:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.662723:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.662733:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.662763:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.662765:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.662767:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.662768:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.662769:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.662770:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.662772:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.662774:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.662776:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.662777:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.662778:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.662779:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.662781:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.662782:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.662785:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.662785:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.662787:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.662789:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232380. 00000020:00000010:6.0:1496184947.662790:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.662792:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.662794:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.662794:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.662795:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.662797:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.662798:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.662799:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.662800:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.662801:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.662802:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.662804:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.662804:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.662805:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.662806:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.662807:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.662808:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662809:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.662810:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662811:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662812:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662813:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.662814:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662815:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.662818:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.662825:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.662825:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.662827:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.662828:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.662829:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.662830:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.662832:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.662832:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.662833:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.662834:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00002000:00000001:6.0:1496184947.662835:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662836:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662838:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662843:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.662845:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.662846:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.662849:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57619 00000100:00000040:6.0:1496184947.662850:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.662851:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671711824 : -131872037839792 : ffff881026075c50) 00000100:00000040:6.0:1496184947.662854:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.662859:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.662860:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.662861:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.662863:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.662864:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.662866:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.662867:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662868:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.662869:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.662870:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.662871:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.662872:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.662873:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.662874:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662875:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.662876:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.662877:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.662879:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77ec00. 02000000:00000001:6.0:1496184947.662880:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662881:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662882:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.662883:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.662885:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.662899:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.662902:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.662903:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.662905:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.662907:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.662908:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.662909:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.662910:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.662912:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.662913:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.662915:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.662916:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.662917:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.662918:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.662918:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.662920:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.662921:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.662922:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.662924:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.662946:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.662952:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662953:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.662956:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662957:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.662958:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.662959:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.662960:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.662962:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7ee00. 00000100:00000010:6.0:1496184947.662963:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.662965:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.662969:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.662970:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.662997:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe520-0x592def0cfe520 00000100:00000001:6.0:1496184947.662999:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.664609:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.664624:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ee00 00000100:00000001:3.0:1496184947.675641:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.675646:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.675647:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ee00 00000100:00000001:3.0:1496184947.686639:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.686702:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.686721:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.686724:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.686727:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.686728:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.686731:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.686732:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.686765:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.686767:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.686769:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.686770:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.686771:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.686772:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.686773:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.686775:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.686775:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.686776:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.686777:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.686779:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.686781:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7ee00. 00000100:00000001:6.0:1496184947.686782:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.686783:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.686784:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77ec00. 02000000:00000001:6.0:1496184947.686785:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.686787:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.686790:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.686792:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.686798:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.686799:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.686800:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.686800:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.686802:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.686804:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.686809:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23949us (24090us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.686814:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57619 00000100:00000040:6.0:1496184947.686816:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.686817:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.686818:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.686821:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.686826:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.686827:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.686828:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.686829:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.686831:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.686832:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.686834:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.686834:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.686835:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.686835:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00002000:00000001:6.0:1496184947.686837:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.686838:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.686840:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232380. 00000020:00000010:6.0:1496184947.686841:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.686843:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.686844:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.686846:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.686848:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.686853:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.686854:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.697381:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.697383:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.697385:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdec50 x1568861104104752 msgsize 520 00000100:00100000:7.0:1496184947.697390:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.697391:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.697393:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.697425:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.697427:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.697429:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.697430:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.697431:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.697432:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.697435:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.697437:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104752 00000020:00000001:6.0:1496184947.697438:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.697439:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.697440:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.697442:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.697443:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.697445:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.697447:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.697447:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.697449:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.697451:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232d00. 00000020:00000010:6.0:1496184947.697453:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.697454:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.697455:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.697456:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.697458:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.697461:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdec50 x1568861104104752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.697466:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.697467:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697472:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.697474:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.697475:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.697476:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.697477:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.697477:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.697478:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.697480:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43365 00000100:00000040:6.0:1496184947.697482:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.697483:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683676240 : -131872025875376 : ffff881026bdec50) 00000100:00000040:6.0:1496184947.697485:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdec50 x1568861104104752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.697490:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.697491:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.697493:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104752:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.697495:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104752 00000020:00000001:6.0:1496184947.697496:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.697498:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.697501:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.697504:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.697504:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.697506:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77ec00. 02000000:00000001:6.0:1496184947.697507:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697508:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.697511:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.697512:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.697514:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.697515:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.697517:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.697519:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.697521:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.697522:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.697523:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.697524:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.697526:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.697526:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.697528:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.697529:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.697530:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.697531:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.697532:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.697533:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.697535:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.697538:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.697538:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.697540:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.697542:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.697542:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.697545:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.697546:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.697547:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.697548:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.697550:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.697551:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.697553:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.697554:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.697555:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.697556:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.697558:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.697559:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.697560:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.697561:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.697562:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.697563:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.697564:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.697566:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.697567:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.697568:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697569:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.697572:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.697574:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.697576:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.697577:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.697579:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.697580:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697581:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.697582:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.697583:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.697585:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.697586:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.697587:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697588:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.697589:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.697590:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.697591:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.697593:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.697593:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.697595:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.697596:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.697597:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.697599:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.697600:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.697601:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.697602:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.697604:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.697605:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.697606:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.697607:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.697608:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.697609:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.697610:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.697611:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104752 00010000:00000001:6.0:1496184947.697613:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.697614:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdec50 x1568861104104752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.697620:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.697621:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.697622:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.697624:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.697626:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.697627:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.697628:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.697629:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.697630:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697631:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.697633:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.697635:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104752, offset 0 00000100:00000001:6.0:1496184947.697650:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.697651:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.697652:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.697653:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.697654:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.697655:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.697656:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.697658:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdec50 x1568861104104752/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.697663:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104752:12345-10.37.248.124@o2ib1:8 Request procesed in 172us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.697668:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43365 00000100:00000040:6.0:1496184947.697669:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.697671:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.697671:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.697673:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232d00. 00000020:00000010:6.0:1496184947.697675:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.697676:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.697677:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.697679:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.697681:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.697684:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.697685:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.698280:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.698283:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.698284:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.698285:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.698287:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.698288:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.698290:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.698292:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.698293:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.698294:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.698295:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.698297:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.698298:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.698300:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.698302:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.698303:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.698304:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.698307:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232480. 00000020:00000010:6.0:1496184947.698308:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.698310:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.698312:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.698313:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.698314:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.698315:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.698316:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.698317:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.698319:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.698320:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.698321:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.698322:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.698323:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.698324:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.698325:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.698326:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.698326:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698327:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.698328:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698329:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698330:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.698332:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.698333:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.698334:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.698337:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.698343:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.698344:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.698346:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.698346:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.698347:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.698349:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.698350:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.698351:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.698351:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.698352:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00002000:00000001:6.0:1496184947.698353:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.698354:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.698357:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.698361:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.698363:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.698364:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.698365:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.698366:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.698367:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.698368:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.698370:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57620 00000100:00000040:6.0:1496184947.698372:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.698373:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671713872 : -131872037837744 : ffff881026076450) 00000100:00000040:6.0:1496184947.698375:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.698380:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.698381:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.698383:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.698385:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.698386:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.698388:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.698388:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698389:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.698390:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.698391:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.698392:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.698393:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.698394:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.698395:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698396:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.698398:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.698398:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.698400:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77ac00. 02000000:00000001:6.0:1496184947.698401:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.698402:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698404:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.698404:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.698406:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.698411:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.698413:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.698414:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.698416:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.698418:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.698419:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.698420:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.698421:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.698423:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.698424:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.698426:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.698428:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.698428:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.698429:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.698430:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.698431:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.698432:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.698433:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.698436:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.698457:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.698463:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698464:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.698467:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.698468:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.698469:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.698470:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.698471:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.698473:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7de00. 00000100:00000010:6.0:1496184947.698474:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.698476:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.698480:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.698481:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.698508:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe540-0x592def0cfe540 00000100:00000001:6.0:1496184947.698510:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.700514:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.700516:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.700522:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.700527:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.700528:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184947.700533:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.700572:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.700583:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.700585:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.700589:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.700590:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.700592:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.700593:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.700627:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.700629:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.700631:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.700632:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.700632:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.700633:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.700635:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.700636:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.700637:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.700637:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.700638:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.700640:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.700642:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7de00. 00000100:00000001:6.0:1496184947.700643:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.700644:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.700645:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77ac00. 02000000:00000001:6.0:1496184947.700646:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.700648:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.700650:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.700652:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.700657:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.700659:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.700659:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.700660:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.700661:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.700663:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.700669:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2287us (2427us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.700674:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57620 00000100:00000040:6.0:1496184947.700676:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.700677:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.700678:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.700681:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.700686:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.700687:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.700689:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.700690:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.700691:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.700692:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.700694:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.700694:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.700695:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.700696:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00002000:00000001:6.0:1496184947.700697:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.700698:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.700700:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232480. 00000020:00000010:6.0:1496184947.700701:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.700703:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.700704:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.700706:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.700708:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.700713:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.700714:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.728944:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.728946:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.728948:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.728949:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.728950:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.728951:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.728954:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.728956:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104784 00000020:00000001:6.0:1496184947.728957:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.728958:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.728959:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.728960:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.728962:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.728963:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.728966:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.728966:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.728968:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.728970:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232280. 00000020:00000010:6.0:1496184947.728971:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.728973:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.728974:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.728975:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.728977:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.728980:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf050 x1568861104104784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.728985:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.728986:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.728990:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.728992:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.728993:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.728994:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.728995:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.728996:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.728997:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.728999:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43366 00000100:00000040:6.0:1496184947.729001:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.729002:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683677264 : -131872025874352 : ffff881026bdf050) 00000100:00000040:6.0:1496184947.729004:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf050 x1568861104104784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.729009:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.729010:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.729012:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104784:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.729014:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104784 00000020:00000001:6.0:1496184947.729015:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.729017:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.729020:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.729023:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.729024:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.729025:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77ac00. 02000000:00000001:6.0:1496184947.729026:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729028:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.729030:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.729032:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.729033:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.729035:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.729036:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.729038:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.729041:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.729042:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.729043:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.729044:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.729045:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.729046:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.729047:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.729048:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.729050:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.729051:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.729052:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.729053:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.729054:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.729057:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.729058:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.729059:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.729061:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.729062:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.729064:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.729065:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.729066:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.729067:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.729069:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.729070:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.729072:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729073:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.729074:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.729075:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.729077:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729078:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.729078:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.729080:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.729081:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.729082:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.729083:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.729084:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.729086:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.729087:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729088:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.729091:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.729093:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.729094:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.729096:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.729098:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.729099:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729100:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.729101:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.729102:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.729104:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.729105:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.729106:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729107:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.729108:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.729109:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.729110:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.729112:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.729112:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.729113:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.729115:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.729116:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.729117:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.729119:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729120:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.729121:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.729123:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.729123:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.729124:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.729125:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.729126:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729127:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.729129:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.729130:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104784 00010000:00000001:6.0:1496184947.729131:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.729133:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf050 x1568861104104784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.729138:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.729139:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.729140:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.729143:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.729144:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.729146:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.729147:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.729148:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.729149:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729150:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.729152:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.729154:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104784, offset 0 00000100:00000001:6.0:1496184947.729164:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729166:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.729167:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.729168:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.729169:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.729170:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729171:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.729172:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf050 x1568861104104784/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.729178:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104784:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (273us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.729182:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43366 00000100:00000040:6.0:1496184947.729184:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.729185:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.729186:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.729188:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232280. 00000020:00000010:6.0:1496184947.729189:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.729191:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.729192:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.729193:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.729195:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.729198:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.729199:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.729255:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.729257:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.729259:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77ac00. 02000000:00000001:7.0:1496184947.729262:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.729262:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.729704:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.729706:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.729708:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.729713:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.729715:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.729717:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.729750:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.729752:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.729754:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.729755:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.729756:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.729757:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.729759:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.729761:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.729763:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.729763:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.729765:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.729766:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.729767:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.729769:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.729771:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.729772:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.729773:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.729775:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232e00. 00000020:00000010:6.0:1496184947.729777:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.729779:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.729780:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.729781:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.729782:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.729783:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.729784:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.729786:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.729787:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.729788:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.729789:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.729790:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.729791:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.729792:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.729793:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.729794:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.729794:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729796:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.729797:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729798:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729799:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729800:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.729801:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729802:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.729805:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.729812:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.729812:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.729814:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.729815:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.729816:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.729817:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.729819:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.729820:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.729820:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.729821:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00002000:00000001:6.0:1496184947.729822:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729823:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729825:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729830:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.729832:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.729833:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.729834:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.729834:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.729835:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.729836:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.729838:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57621 00000100:00000040:6.0:1496184947.729840:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.729841:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671704656 : -131872037846960 : ffff881026074050) 00000100:00000040:6.0:1496184947.729843:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.729849:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.729849:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.729851:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.729853:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.729854:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.729856:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.729857:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729858:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.729859:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.729860:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.729861:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.729862:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.729863:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.729863:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729865:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.729866:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.729867:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.729869:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77e400. 02000000:00000001:6.0:1496184947.729870:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729871:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729872:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.729873:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.729875:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.729879:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.729882:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.729882:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.729885:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.729886:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.729887:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.729888:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.729901:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.729904:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.729904:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.729906:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.729908:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.729909:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.729909:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.729910:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.729912:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.729913:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.729914:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.729917:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.729938:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.729944:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729945:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.729948:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729949:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.729950:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.729951:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.729952:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.729954:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e000. 00000100:00000010:6.0:1496184947.729955:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.729957:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.729961:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.729962:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.729988:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe560-0x592def0cfe560 00000100:00000001:6.0:1496184947.729990:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.731471:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.731473:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184947.731482:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.731498:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.731498:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184947.731503:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.731542:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.731549:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.731554:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.731558:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.731559:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.731561:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.731563:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.731596:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.731598:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.731600:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.731601:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.731602:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.731604:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.731605:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.731606:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.731607:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.731607:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.731609:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.731610:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.731612:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e000. 00000100:00000001:6.0:1496184947.731613:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.731614:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.731615:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77e400. 02000000:00000001:6.0:1496184947.731616:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.731618:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.731620:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.731622:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.731627:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.731629:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.731629:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.731630:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.731631:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.731633:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.731639:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1789us (1929us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.731644:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57621 00000100:00000040:6.0:1496184947.731646:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.731647:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.731648:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.731651:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.731656:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.731657:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.731658:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.731659:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.731661:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.731662:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.731664:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.731664:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.731665:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.731665:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00002000:00000001:6.0:1496184947.731667:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.731668:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.731669:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232e00. 00000020:00000010:6.0:1496184947.731671:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.731673:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.731674:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.731675:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.731678:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.731683:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.731684:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.760352:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.760353:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.760356:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf450 x1568861104104816 msgsize 520 00000100:00100000:7.0:1496184947.760361:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.760362:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.760364:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.760395:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.760398:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.760399:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.760400:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.760401:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.760402:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.760405:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.760406:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104816 00000020:00000001:6.0:1496184947.760408:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.760409:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.760410:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.760411:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.760413:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.760414:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.760417:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.760418:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.760419:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.760422:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232a80. 00000020:00000010:6.0:1496184947.760423:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.760425:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.760426:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.760427:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.760429:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.760432:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf450 x1568861104104816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.760437:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.760438:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760442:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.760444:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.760445:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.760446:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.760447:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.760448:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.760448:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.760451:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43367 00000100:00000040:6.0:1496184947.760452:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.760453:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683678288 : -131872025873328 : ffff881026bdf450) 00000100:00000040:6.0:1496184947.760456:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf450 x1568861104104816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.760461:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.760462:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.760464:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104816:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.760466:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104816 00000020:00000001:6.0:1496184947.760467:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.760469:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.760472:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.760474:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.760476:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.760477:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77e400. 02000000:00000001:6.0:1496184947.760478:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760479:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.760482:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.760483:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.760485:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.760487:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.760488:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.760490:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.760492:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.760494:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.760495:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.760496:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.760497:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.760497:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.760499:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.760500:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.760501:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.760502:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.760503:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.760504:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.760506:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.760509:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.760510:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.760511:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.760513:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.760513:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.760515:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.760517:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.760517:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.760519:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.760520:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.760522:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.760523:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.760524:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.760525:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.760527:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.760528:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.760529:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.760530:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.760532:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.760533:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.760534:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.760535:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.760536:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.760538:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.760539:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760540:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.760543:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.760545:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.760546:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.760548:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.760549:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.760550:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760551:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.760552:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.760554:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.760555:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.760556:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.760558:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760559:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.760560:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.760561:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.760562:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.760563:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.760564:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.760565:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.760567:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.760568:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.760569:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.760570:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.760572:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.760573:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.760575:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.760575:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.760576:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.760577:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.760578:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.760579:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.760580:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.760581:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104816 00010000:00000001:6.0:1496184947.760583:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.760585:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf450 x1568861104104816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.760590:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.760591:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.760592:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.760594:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.760596:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.760597:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.760598:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.760600:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.760601:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760602:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.760603:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.760606:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104816, offset 0 00000100:00000001:6.0:1496184947.760617:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.760618:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.760619:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.760620:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.760621:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.760622:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.760623:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.760624:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf450 x1568861104104816/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.760630:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104816:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (272us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.760634:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43367 00000100:00000040:6.0:1496184947.760636:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.760637:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.760638:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.760639:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232a80. 00000020:00000010:6.0:1496184947.760641:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.760643:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.760644:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.760646:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.760647:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.760651:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.760652:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.760701:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.760703:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.760705:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77e400. 02000000:00000001:7.0:1496184947.760708:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.760709:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.761195:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.761197:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.761199:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.761204:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.761205:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.761208:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.761239:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.761241:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.761243:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.761244:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.761245:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.761246:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.761249:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.761250:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.761252:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.761253:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.761254:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.761255:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.761257:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.761258:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.761260:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.761261:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.761263:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.761265:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26200. 00000020:00000010:6.0:1496184947.761266:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.761269:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.761270:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.761271:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.761272:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.761273:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.761274:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.761276:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.761277:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.761278:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.761279:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.761280:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.761281:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.761282:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.761283:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.761284:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.761284:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761285:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.761287:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761288:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761289:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.761290:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.761291:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.761292:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.761295:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.761301:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.761302:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.761304:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.761305:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.761306:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.761307:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.761309:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.761310:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.761310:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.761311:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00002000:00000001:6.0:1496184947.761312:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.761313:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.761315:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.761320:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.761322:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.761323:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.761324:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.761324:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.761325:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.761326:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.761328:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57622 00000100:00000040:6.0:1496184947.761330:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.761331:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671707728 : -131872037843888 : ffff881026074c50) 00000100:00000040:6.0:1496184947.761333:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.761339:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.761339:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.761341:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.761343:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.761344:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.761346:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.761347:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761348:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.761348:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.761350:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.761351:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.761352:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.761353:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.761353:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761354:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.761356:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.761357:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.761358:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77bc00. 02000000:00000001:6.0:1496184947.761359:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.761361:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761362:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.761363:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.761365:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.761369:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.761371:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.761372:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.761374:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.761376:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.761377:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.761378:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.761380:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.761381:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.761382:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.761384:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.761386:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.761386:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.761387:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.761388:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.761390:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.761390:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.761391:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.761394:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.761417:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.761423:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761424:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.761427:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.761429:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.761430:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.761431:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.761432:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.761433:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e400. 00000100:00000010:6.0:1496184947.761435:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.761436:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.761440:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.761441:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.761467:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe580-0x592def0cfe580 00000100:00000001:6.0:1496184947.761470:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.762836:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.762839:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.762859:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.762863:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.762864:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.762869:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.762908:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.762922:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.762924:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.762928:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.762929:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.762931:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.762933:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.762966:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.762968:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.762970:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.762971:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.762972:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.762973:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.762974:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.762976:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.762976:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.762977:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.762978:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.762980:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.762981:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e400. 00000100:00000001:6.0:1496184947.762983:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.762984:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.762985:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77bc00. 02000000:00000001:6.0:1496184947.762986:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.762988:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.762990:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.762992:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.762997:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.762999:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.763000:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.763000:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.763002:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.763003:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.763009:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1669us (1808us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.763014:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57622 00000100:00000040:6.0:1496184947.763016:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.763017:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.763018:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.763021:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.763026:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.763027:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.763028:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.763029:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.763030:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.763032:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.763033:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.763034:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.763035:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.763035:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00002000:00000001:6.0:1496184947.763037:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.763038:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.763039:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26200. 00000020:00000010:6.0:1496184947.763041:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.763043:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.763044:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.763045:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.763048:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.763053:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.763054:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.791098:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.791101:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.791102:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.791103:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.791104:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.791105:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.791108:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.791110:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104848 00000020:00000001:6.0:1496184947.791111:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.791112:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.791113:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.791114:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.791116:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.791118:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.791120:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.791120:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.791122:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.791124:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26b80. 00000020:00000010:6.0:1496184947.791125:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.791127:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.791128:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.791129:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.791131:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.791134:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf850 x1568861104104848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.791139:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.791140:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791145:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.791147:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.791147:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.791148:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.791149:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.791150:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.791151:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.791153:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43368 00000100:00000040:6.0:1496184947.791155:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.791156:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683679312 : -131872025872304 : ffff881026bdf850) 00000100:00000040:6.0:1496184947.791158:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf850 x1568861104104848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.791163:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.791164:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.791166:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104848:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.791168:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104848 00000020:00000001:6.0:1496184947.791169:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.791171:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.791174:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.791176:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.791177:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.791179:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77bc00. 02000000:00000001:6.0:1496184947.791180:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791181:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.791184:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.791185:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.791187:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.791188:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.791190:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.791192:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.791195:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.791196:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.791197:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.791198:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.791199:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.791200:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.791201:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.791202:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.791204:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.791204:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.791205:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.791206:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.791208:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.791211:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.791211:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.791213:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.791215:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.791215:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.791217:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.791219:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.791219:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.791220:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.791222:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.791224:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.791225:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.791226:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.791227:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.791229:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.791230:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.791231:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.791232:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.791234:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.791235:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.791236:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.791237:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.791238:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.791240:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.791241:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791242:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.791245:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.791247:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.791248:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.791250:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.791251:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.791252:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791253:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.791254:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.791255:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.791257:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.791258:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.791259:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791260:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.791262:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.791262:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.791263:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.791265:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.791265:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.791267:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.791268:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.791270:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.791271:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.791272:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791274:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.791275:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.791276:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.791277:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.791278:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.791279:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.791279:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791281:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.791282:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.791283:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104848 00010000:00000001:6.0:1496184947.791285:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.791286:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf850 x1568861104104848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.791292:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.791293:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.791294:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.791296:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.791298:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.791299:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.791300:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.791301:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.791302:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791303:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.791305:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.791307:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104848, offset 0 00000100:00000001:6.0:1496184947.791317:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791319:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.791320:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.791321:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.791322:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.791323:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791324:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.791325:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf850 x1568861104104848/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.791335:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104848:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.791339:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43368 00000100:00000040:6.0:1496184947.791341:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.791342:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.791343:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.791344:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26b80. 00000020:00000010:6.0:1496184947.791346:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.791347:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.791349:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.791350:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.791352:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.791355:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.791356:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.791899:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.791902:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.791903:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.791904:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.791905:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.791906:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.791909:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.791911:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.791912:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.791913:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.791914:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.791915:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.791917:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.791919:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.791921:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.791921:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.791923:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.791925:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26c80. 00000020:00000010:6.0:1496184947.791927:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.791929:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.791930:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.791939:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.791940:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.791941:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.791942:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.791944:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.791945:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.791946:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.791947:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.791948:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.791949:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.791950:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.791951:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.791952:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.791952:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791953:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.791955:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791956:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.791957:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.791958:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.791959:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.791960:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.791963:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.791969:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.791970:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.791972:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.791973:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.791974:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.791975:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.791977:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.791977:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.791978:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.791978:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00002000:00000001:6.0:1496184947.791980:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791981:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791983:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.791988:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.791990:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.791990:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.791991:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.791992:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.791993:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.791994:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.791996:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57623 00000100:00000040:6.0:1496184947.791998:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.791999:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671700560 : -131872037851056 : ffff881026073050) 00000100:00000040:6.0:1496184947.792001:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.792006:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.792007:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.792009:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.792011:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.792012:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.792014:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.792015:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.792016:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.792016:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.792018:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.792019:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.792020:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.792021:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.792021:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.792022:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.792024:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.792025:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.792026:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a400. 02000000:00000001:6.0:1496184947.792027:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.792029:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.792030:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.792031:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.792033:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.792037:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.792039:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.792040:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.792043:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.792044:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.792045:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.792046:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.792047:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.792049:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.792050:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.792052:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.792054:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.792054:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.792055:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.792056:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.792063:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.792063:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.792064:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.792067:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.792089:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.792095:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.792096:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.792098:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.792100:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.792101:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.792102:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.792103:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.792104:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c200. 00000100:00000010:6.0:1496184947.792106:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.792107:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.792112:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.792113:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.792138:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe5a0-0x592def0cfe5a0 00000100:00000001:6.0:1496184947.792141:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.793499:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.793501:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184947.804628:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.804633:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.804634:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184947.815627:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.815706:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.815712:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.815715:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.815718:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.815719:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.815721:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.815723:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.815756:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.815758:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.815760:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.815761:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.815761:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.815763:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.815764:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.815765:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.815766:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.815767:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.815768:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.815770:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.815771:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c200. 00000100:00000001:6.0:1496184947.815773:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.815773:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.815775:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a400. 02000000:00000001:6.0:1496184947.815776:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.815778:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.815779:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.815782:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.815787:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.815789:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.815789:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.815790:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.815791:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.815793:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.815799:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23791us (23936us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.815804:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57623 00000100:00000040:6.0:1496184947.815805:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.815807:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.815808:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.815810:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.815816:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.815816:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.815818:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.815819:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.815820:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.815822:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.815823:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.815824:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.815824:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.815825:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00002000:00000001:6.0:1496184947.815826:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.815828:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.815829:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26c80. 00000020:00000010:6.0:1496184947.815831:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.815832:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.815834:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.815835:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.815837:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.815843:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.815844:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.825971:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.825973:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.825984:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c9050 x1568861104104880 msgsize 520 00000100:00100000:7.0:1496184947.825989:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.825990:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.825992:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.826024:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.826027:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.826029:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.826030:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.826031:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.826032:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.826034:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.826036:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104880 00000020:00000001:6.0:1496184947.826038:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.826039:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.826040:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.826041:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.826042:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.826044:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.826046:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.826047:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.826048:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7f200. 00000020:00000010:6.0:1496184947.826050:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26180. 00000020:00000010:6.0:1496184947.826052:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.826054:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.826054:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.826055:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.826057:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.826060:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c9050 x1568861104104880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.826065:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.826066:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826071:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.826073:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.826074:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.826075:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.826076:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.826076:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.826077:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.826080:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43369 00000100:00000040:6.0:1496184947.826081:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.826082:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677295696 : -131872032255920 : ffff8810265c9050) 00000100:00000040:6.0:1496184947.826085:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c9050 x1568861104104880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.826090:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.826091:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.826092:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104880:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.826094:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104880 00000020:00000001:6.0:1496184947.826096:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.826098:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.826100:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.826103:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.826104:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.826105:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a400. 02000000:00000001:6.0:1496184947.826107:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826108:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.826110:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.826112:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.826114:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.826115:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.826117:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.826119:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.826121:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.826122:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.826123:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.826124:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.826125:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.826126:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.826127:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.826128:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.826130:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.826131:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.826132:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.826133:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.826134:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.826137:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.826138:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.826140:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.826141:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.826142:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.826144:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.826146:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.826146:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.826147:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.826149:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.826151:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.826152:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.826153:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.826154:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.826156:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.826157:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.826158:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.826159:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.826160:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.826162:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.826163:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.826164:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.826165:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.826166:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.826168:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826169:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.826172:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.826174:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.826175:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.826177:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.826178:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.826179:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826180:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.826182:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.826183:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.826184:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.826185:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.826187:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826188:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.826189:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.826189:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.826190:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.826192:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.826193:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.826194:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.826196:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.826197:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.826198:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.826199:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826201:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.826202:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.826203:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.826204:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.826205:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.826206:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.826207:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826208:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.826210:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.826211:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104880 00010000:00000001:6.0:1496184947.826212:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.826214:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c9050 x1568861104104880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.826220:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.826220:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.826222:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.826224:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.826225:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.826227:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.826228:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.826229:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.826230:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826231:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.826233:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.826235:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104880, offset 0 00000100:00000001:6.0:1496184947.826246:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826247:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.826249:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.826250:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.826251:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.826251:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826252:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.826254:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c9050 x1568861104104880/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.826260:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104880:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (274us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.826264:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43369 00000100:00000040:6.0:1496184947.826265:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.826267:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.826268:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.826269:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26180. 00000020:00000010:6.0:1496184947.826271:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.826272:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7f200. 00000020:00000040:6.0:1496184947.826273:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.826275:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.826277:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.826280:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.826281:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.826855:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.826858:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.826860:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.826861:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.826862:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.826863:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.826865:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.826867:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.826869:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.826869:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.826871:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.826872:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.826873:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.826875:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.826877:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.826878:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.826879:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7f200. 00000020:00000010:6.0:1496184947.826881:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26500. 00000020:00000010:6.0:1496184947.826883:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.826885:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.826886:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.826887:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.826888:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.826889:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.826903:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.826904:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.826906:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.826907:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.826908:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.826909:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.826910:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.826912:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.826912:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.826913:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.826914:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826915:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.826916:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826917:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826918:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.826919:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.826920:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.826922:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.826924:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.826931:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.826932:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.826933:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.826934:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.826935:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.826937:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.826938:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.826939:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.826939:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.826940:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00002000:00000001:6.0:1496184947.826941:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826943:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826945:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826949:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.826951:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.826952:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.826953:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.826953:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.826954:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.826955:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.826958:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57624 00000100:00000040:6.0:1496184947.826960:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.826961:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671702608 : -131872037849008 : ffff881026073850) 00000100:00000040:6.0:1496184947.826963:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.826968:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.826969:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.826971:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.826973:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.826974:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.826976:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.826976:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826977:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.826978:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.826980:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.826981:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.826982:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.826982:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.826983:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826984:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.826986:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.826987:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.826988:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e778c00. 02000000:00000001:6.0:1496184947.826990:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.826991:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.826992:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.826993:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.826995:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.826999:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.827002:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.827002:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.827005:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.827007:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.827008:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.827009:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.827010:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.827012:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.827013:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.827015:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.827016:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.827017:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.827017:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.827018:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.827020:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.827021:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.827022:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.827024:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.827045:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.827051:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.827052:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.827055:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.827056:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.827057:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.827058:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.827060:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.827061:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d400. 00000100:00000010:6.0:1496184947.827062:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.827064:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.827068:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.827069:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.827096:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe5c0-0x592def0cfe5c0 00000100:00000001:6.0:1496184947.827098:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.828888:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.828913:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184947.839973:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.839978:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.839979:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184947.850962:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.851026:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.851044:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.851047:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.851050:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.851051:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.851053:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.851055:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.851088:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.851090:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.851092:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.851093:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.851094:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.851095:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.851096:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.851098:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.851098:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.851099:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.851100:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.851102:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.851104:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d400. 00000100:00000001:6.0:1496184947.851105:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.851106:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.851107:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e778c00. 02000000:00000001:6.0:1496184947.851108:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.851110:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.851112:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.851114:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.851119:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.851121:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.851122:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.851122:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.851123:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.851125:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.851136:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24166us (24319us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.851141:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57624 00000100:00000040:6.0:1496184947.851142:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.851144:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.851145:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.851147:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.851153:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.851153:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.851155:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.851156:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.851157:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.851159:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.851160:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.851161:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.851161:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.851162:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00002000:00000001:6.0:1496184947.851163:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.851165:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.851166:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26500. 00000020:00000010:6.0:1496184947.851168:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.851169:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7f200. 00000020:00000040:6.0:1496184947.851171:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.851172:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.851174:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.851179:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.851180:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.861702:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.861704:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.861707:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8c50 x1568861104104912 msgsize 520 00000100:00100000:7.0:1496184947.861712:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.861713:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.861715:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.861749:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.861751:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.861753:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.861754:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.861755:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.861756:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.861759:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.861760:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104912 00000020:00000001:6.0:1496184947.861762:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.861763:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.861764:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.861765:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.861767:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.861768:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.861770:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.861771:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.861773:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.861775:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26300. 00000020:00000010:6.0:1496184947.861776:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.861778:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.861779:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.861780:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.861782:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.861785:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8c50 x1568861104104912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.861790:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.861791:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861795:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.861797:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.861798:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.861799:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.861800:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.861801:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.861802:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.861804:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43370 00000100:00000040:6.0:1496184947.861805:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.861807:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677294672 : -131872032256944 : ffff8810265c8c50) 00000100:00000040:6.0:1496184947.861809:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8c50 x1568861104104912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.861814:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.861815:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.861817:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104912:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.861819:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104912 00000020:00000001:6.0:1496184947.861820:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.861822:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.861825:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.861828:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.861829:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.861830:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778c00. 02000000:00000001:6.0:1496184947.861831:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861832:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.861835:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.861836:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.861838:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.861840:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.861841:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.861843:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.861846:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.861847:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.861848:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.861849:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.861850:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.861850:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.861852:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.861853:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.861854:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.861855:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.861856:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.861857:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.861859:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.861862:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.861863:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.861864:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.861866:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.861867:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.861869:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.861870:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.861871:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.861872:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.861874:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.861875:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.861876:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.861877:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.861878:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.861880:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.861881:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.861882:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.861883:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.861885:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.861886:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.861887:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.861888:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.861901:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.861903:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.861904:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861905:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.861908:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.861910:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.861911:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.861913:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.861915:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.861916:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861917:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.861918:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.861919:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.861921:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.861922:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.861923:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861924:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.861925:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.861926:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.861927:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.861929:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.861929:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.861931:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.861932:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.861933:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.861935:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.861936:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.861937:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.861938:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.861940:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.861941:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.861942:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.861942:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.861943:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.861944:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.861946:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.861947:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104912 00010000:00000001:6.0:1496184947.861948:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.861950:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8c50 x1568861104104912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.861955:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.861956:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.861958:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.861960:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.861961:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.861963:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.861964:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.861965:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.861966:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861967:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.861969:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.861971:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104912, offset 0 00000100:00000001:6.0:1496184947.861982:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.861983:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.861984:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.861985:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.861986:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.861987:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.861988:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.861990:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8c50 x1568861104104912/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.861995:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104912:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (287us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.862000:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43370 00000100:00000040:6.0:1496184947.862001:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.862003:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.862003:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.862005:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26300. 00000020:00000010:6.0:1496184947.862006:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.862008:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.862009:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.862011:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.862013:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.862016:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.862017:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.862087:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.862090:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.862093:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e778c00. 02000000:00000001:7.0:1496184947.862097:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.862098:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.862553:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.862555:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.862558:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.862563:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.862571:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.862573:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.862605:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.862607:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.862609:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.862610:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.862611:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.862612:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.862615:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.862617:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.862618:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.862619:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.862620:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.862621:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.862623:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.862624:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.862627:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.862627:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.862629:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000010:6.0:1496184947.862631:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26080. 00000020:00000010:6.0:1496184947.862632:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.862635:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.862636:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.862637:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.862638:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.862639:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.862640:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.862642:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.862643:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.862644:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.862645:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.862647:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.862647:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.862649:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.862649:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.862650:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.862651:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862652:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.862653:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862654:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862655:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.862656:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.862657:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.862659:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.862662:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.862668:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.862669:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.862670:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.862671:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.862672:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.862674:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.862675:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.862676:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.862676:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.862677:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00002000:00000001:6.0:1496184947.862678:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.862679:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.862681:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.862686:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.862688:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.862689:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.862690:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.862690:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.862691:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.862692:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.862695:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57625 00000100:00000040:6.0:1496184947.862696:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.862697:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671695440 : -131872037856176 : ffff881026071c50) 00000100:00000040:6.0:1496184947.862700:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.862705:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.862706:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.862708:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.862710:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.862711:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.862713:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.862713:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862714:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.862715:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.862717:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.862718:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.862719:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.862719:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.862720:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862721:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.862723:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.862724:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.862725:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77fc00. 02000000:00000001:6.0:1496184947.862726:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.862728:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862729:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.862730:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.862732:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.862736:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.862738:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.862739:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.862741:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.862743:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.862744:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.862745:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.862746:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.862748:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.862749:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.862751:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.862753:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.862753:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.862754:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.862755:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.862757:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.862757:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.862758:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.862761:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.862783:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.862789:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862790:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.862793:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.862794:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.862795:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.862796:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.862797:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.862798:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e400. 00000100:00000010:6.0:1496184947.862800:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.862801:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.862806:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.862807:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.862833:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe5e0-0x592def0cfe5e0 00000100:00000001:6.0:1496184947.862835:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.864456:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.864459:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.875537:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.875542:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.875543:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e400 00000100:00000001:3.0:1496184947.886533:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.886605:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.886612:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.886614:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.886617:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.886618:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.886621:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.886622:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.886655:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.886657:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.886659:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.886660:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.886661:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.886663:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.886664:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.886665:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.886666:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.886666:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.886668:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.886669:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.886671:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e400. 00000100:00000001:6.0:1496184947.886672:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.886673:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.886674:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77fc00. 02000000:00000001:6.0:1496184947.886676:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.886677:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.886679:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.886681:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.886687:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.886688:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.886689:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.886689:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.886691:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.886693:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.886698:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23992us (24139us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.886704:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57625 00000100:00000040:6.0:1496184947.886705:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.886707:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.886708:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.886710:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.886716:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.886717:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.886718:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.886719:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.886720:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.886722:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.886723:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.886724:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.886725:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.886725:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00002000:00000001:6.0:1496184947.886727:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.886728:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.886729:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26080. 00000020:00000010:6.0:1496184947.886731:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.886732:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7d400. 00000020:00000040:6.0:1496184947.886734:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.886735:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.886737:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.886742:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.886743:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.897366:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.897369:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.897370:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.897371:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.897372:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.897373:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.897376:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.897378:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104944 00000020:00000001:6.0:1496184947.897379:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.897380:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.897381:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.897383:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.897384:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.897386:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.897388:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.897389:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.897390:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.897392:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26600. 00000020:00000010:6.0:1496184947.897394:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.897395:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.897396:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.897397:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.897399:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.897402:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8050 x1568861104104944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.897407:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.897408:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897413:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.897415:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.897416:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.897416:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.897417:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.897418:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.897419:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.897421:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43371 00000100:00000040:6.0:1496184947.897423:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.897424:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677291600 : -131872032260016 : ffff8810265c8050) 00000100:00000040:6.0:1496184947.897426:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8050 x1568861104104944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.897431:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.897432:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.897434:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104944:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.897436:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104944 00000020:00000001:6.0:1496184947.897437:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.897439:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.897442:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.897444:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.897445:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.897447:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77fc00. 02000000:00000001:6.0:1496184947.897448:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897449:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.897452:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.897453:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.897455:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.897457:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.897458:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.897460:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.897463:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.897464:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.897465:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.897466:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.897467:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.897467:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.897469:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.897470:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.897471:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.897472:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.897473:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.897474:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.897476:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.897479:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.897480:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.897481:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.897483:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.897484:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.897486:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.897487:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.897488:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.897489:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.897491:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.897492:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.897494:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.897495:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.897495:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.897497:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.897499:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.897499:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.897500:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.897502:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.897503:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.897504:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.897505:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.897506:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.897508:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.897509:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897510:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.897513:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.897515:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.897516:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.897518:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.897519:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.897520:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897521:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.897523:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.897524:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.897525:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.897526:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.897528:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897529:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.897530:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.897531:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.897532:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.897533:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.897534:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.897535:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.897537:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.897538:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.897539:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.897540:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.897542:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.897543:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.897544:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.897545:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.897546:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.897547:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.897548:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.897549:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.897550:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.897551:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104944 00010000:00000001:6.0:1496184947.897553:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.897555:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8050 x1568861104104944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.897560:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.897561:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.897562:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.897564:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.897566:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.897567:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.897568:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.897570:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.897570:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897572:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.897573:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.897575:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104944, offset 0 00000100:00000001:6.0:1496184947.897591:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.897592:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.897594:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.897595:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.897596:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.897596:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.897597:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.897599:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8050 x1568861104104944/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.897605:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104944:12345-10.37.248.124@o2ib1:8 Request procesed in 172us (284us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.897609:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43371 00000100:00000040:6.0:1496184947.897611:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.897612:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.897613:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.897614:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26600. 00000020:00000010:6.0:1496184947.897616:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.897617:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.897619:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.897620:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.897622:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.897625:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.897626:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.898239:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.898241:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.898243:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.898244:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.898245:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.898246:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.898248:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.898250:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.898252:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.898253:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.898254:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.898255:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.898257:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.898258:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.898261:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.898261:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.898263:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.898265:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26900. 00000020:00000010:6.0:1496184947.898267:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.898269:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.898270:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.898271:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.898272:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.898273:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.898274:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.898275:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.898276:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.898278:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.898279:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.898280:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.898281:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.898282:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.898283:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.898283:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.898284:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898285:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.898286:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898287:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898288:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.898290:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.898291:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.898292:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.898295:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.898301:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.898302:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.898304:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.898304:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.898305:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.898307:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.898308:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.898309:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.898310:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.898310:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00002000:00000001:6.0:1496184947.898312:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.898313:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.898315:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.898319:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.898321:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.898322:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.898323:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.898324:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.898325:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.898326:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.898328:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57626 00000100:00000040:6.0:1496184947.898330:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.898331:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671701584 : -131872037850032 : ffff881026073450) 00000100:00000040:6.0:1496184947.898333:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.898339:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.898339:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.898341:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.898343:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.898344:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.898346:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.898347:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898348:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.898348:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.898350:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.898351:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.898352:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.898353:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.898353:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898355:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.898356:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.898357:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.898359:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184947.898360:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.898361:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898362:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.898363:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.898365:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.898369:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.898372:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.898372:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.898375:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.898376:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.898377:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.898378:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.898380:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.898382:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.898382:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.898384:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.898386:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.898386:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.898387:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.898388:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.898390:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.898390:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.898391:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.898394:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.898415:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.898421:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898422:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.898425:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.898426:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.898427:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.898428:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.898429:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.898430:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e600. 00000100:00000010:6.0:1496184947.898432:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.898433:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.898438:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.898439:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.898469:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe600-0x592def0cfe600 00000100:00000001:6.0:1496184947.898471:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.900104:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.900107:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184947.900127:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.900132:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.900132:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184947.900137:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.900174:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.900181:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.900183:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.900187:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.900188:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.900190:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.900191:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.900225:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.900227:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.900229:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.900230:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.900230:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.900232:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.900233:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.900234:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.900235:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.900236:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.900237:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.900238:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.900240:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e600. 00000100:00000001:6.0:1496184947.900242:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.900242:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.900244:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184947.900245:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.900247:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.900248:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.900251:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.900256:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.900258:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.900258:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.900259:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.900260:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.900262:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.900268:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1928us (2073us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.900273:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57626 00000100:00000040:6.0:1496184947.900275:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.900276:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.900277:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.900280:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.900285:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.900286:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.900287:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.900288:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.900289:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.900291:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.900292:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.900293:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.900294:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.900294:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00002000:00000001:6.0:1496184947.900296:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.900297:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.900298:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26900. 00000020:00000010:6.0:1496184947.900300:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.900301:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.900303:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.900304:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.900306:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.900312:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.900313:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.929209:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.929212:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.929213:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.929214:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.929215:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.929216:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.929219:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.929221:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104104976 00000020:00000001:6.0:1496184947.929222:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.929223:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.929224:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.929225:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.929227:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.929228:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.929230:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.929231:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.929233:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.929235:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26000. 00000020:00000010:6.0:1496184947.929236:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.929238:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.929239:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.929240:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.929242:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.929246:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7450 x1568861104104976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.929251:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.929251:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929256:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.929258:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.929259:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.929260:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.929260:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.929261:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.929262:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.929264:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43372 00000100:00000040:6.0:1496184947.929266:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.929267:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676239952 : -131872033311664 : ffff8810264c7450) 00000100:00000040:6.0:1496184947.929269:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7450 x1568861104104976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.929275:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.929275:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.929277:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104976:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.929279:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104104976 00000020:00000001:6.0:1496184947.929280:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.929282:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.929285:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.929288:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.929289:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.929290:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77f000. 02000000:00000001:6.0:1496184947.929291:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929293:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.929295:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.929297:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.929298:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.929300:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.929301:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.929303:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.929306:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.929307:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.929308:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.929309:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.929310:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.929311:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.929312:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.929314:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.929315:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.929316:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.929317:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.929318:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.929319:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.929322:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.929323:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.929325:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.929326:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.929327:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.929329:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.929331:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.929331:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.929332:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.929334:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.929336:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.929337:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.929338:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.929339:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.929340:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.929342:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.929343:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.929344:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.929346:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.929347:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.929348:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.929349:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.929350:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.929352:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.929353:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929354:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.929356:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.929359:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.929360:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.929362:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.929363:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.929364:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929365:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.929366:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.929367:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.929369:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.929370:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.929371:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929373:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.929374:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.929374:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.929375:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.929377:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.929377:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.929379:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.929380:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.929382:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.929383:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.929384:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.929386:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.929387:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.929388:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.929389:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.929390:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.929391:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.929392:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.929393:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.929394:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.929395:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104104976 00010000:00000001:6.0:1496184947.929397:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.929398:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7450 x1568861104104976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.929404:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.929405:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.929406:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.929408:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.929410:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.929411:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.929412:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.929413:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.929414:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929415:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.929417:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.929419:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104104976, offset 0 00000100:00000001:6.0:1496184947.929430:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.929431:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.929432:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.929433:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.929434:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.929435:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.929436:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.929438:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7450 x1568861104104976/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.929447:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104104976:12345-10.37.248.124@o2ib1:8 Request procesed in 171us (275us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.929452:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43372 00000100:00000040:6.0:1496184947.929453:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.929454:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.929455:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.929457:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26000. 00000020:00000010:6.0:1496184947.929458:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.929460:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.929461:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.929462:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.929464:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.929467:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.929468:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.930023:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.930025:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.930027:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.930028:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.930029:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.930030:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.930032:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.930034:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.930036:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.930036:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.930037:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.930039:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.930040:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.930042:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.930044:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.930045:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.930046:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.930048:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26700. 00000020:00000010:6.0:1496184947.930050:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.930052:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.930053:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.930054:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.930055:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.930056:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.930057:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.930059:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.930060:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.930061:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.930062:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.930063:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.930064:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.930065:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.930066:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.930067:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.930067:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930068:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.930070:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930071:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930072:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.930073:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.930074:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.930075:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.930078:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.930084:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.930085:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.930087:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.930088:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.930088:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.930090:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.930091:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.930092:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.930093:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.930093:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00002000:00000001:6.0:1496184947.930095:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.930096:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.930098:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.930103:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.930105:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.930105:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.930106:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.930107:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.930108:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.930109:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.930111:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57627 00000100:00000040:6.0:1496184947.930113:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.930114:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671697488 : -131872037854128 : ffff881026072450) 00000100:00000040:6.0:1496184947.930116:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.930122:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.930122:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.930124:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.930126:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.930127:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.930129:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.930130:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930131:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.930132:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.930133:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.930134:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.930135:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.930136:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.930136:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930137:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.930139:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.930140:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.930141:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e779800. 02000000:00000001:6.0:1496184947.930142:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.930144:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930145:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.930146:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.930148:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.930152:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.930155:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.930156:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.930158:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.930160:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.930161:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.930162:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.930163:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.930165:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.930166:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.930168:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.930169:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.930170:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.930171:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.930172:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.930173:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.930174:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.930175:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.930177:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.930199:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.930204:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930206:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.930208:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.930210:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.930211:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.930212:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.930213:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.930214:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7da00. 00000100:00000010:6.0:1496184947.930215:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.930217:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.930221:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.930222:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.930249:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe620-0x592def0cfe620 00000100:00000001:6.0:1496184947.930252:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.931720:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.931723:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00000001:3.0:1496184947.931743:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.931748:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.931748:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00000001:3.0:1496184947.931753:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.931792:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.931802:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.931804:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.931808:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.931809:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.931811:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.931813:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.931846:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.931848:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.931850:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.931851:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.931852:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.931853:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.931854:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.931856:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.931857:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.931857:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.931858:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.931860:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.931862:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7da00. 00000100:00000001:6.0:1496184947.931863:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.931864:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.931865:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779800. 02000000:00000001:6.0:1496184947.931866:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.931868:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.931870:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.931872:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.931878:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.931879:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.931880:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.931881:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.931882:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.931884:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.931904:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1767us (1908us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.931909:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57627 00000100:00000040:6.0:1496184947.931911:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.931912:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.931913:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.931916:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.931922:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.931922:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.931924:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.931925:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.931926:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.931928:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.931929:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.931930:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.931930:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.931931:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00002000:00000001:6.0:1496184947.931932:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.931933:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.931935:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26700. 00000020:00000010:6.0:1496184947.931936:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.931938:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.931940:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.931941:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.931944:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.931954:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.931955:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.960549:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.960551:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184947.960553:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7c50 x1568861104105008 msgsize 520 00000100:00100000:7.0:1496184947.960558:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.960559:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.960561:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.960592:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.960595:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.960596:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.960597:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.960598:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.960599:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.960602:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.960604:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105008 00000020:00000001:6.0:1496184947.960605:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.960606:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.960607:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.960608:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.960610:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.960611:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.960614:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.960614:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.960616:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.960618:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26380. 00000020:00000010:6.0:1496184947.960619:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.960621:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.960622:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.960623:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.960625:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.960628:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7c50 x1568861104105008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.960633:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.960634:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960638:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.960640:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.960641:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.960642:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.960643:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.960644:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.960644:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.960647:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43373 00000100:00000040:6.0:1496184947.960648:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.960649:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676242000 : -131872033309616 : ffff8810264c7c50) 00000100:00000040:6.0:1496184947.960652:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7c50 x1568861104105008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.960657:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.960658:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.960660:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105008:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.960662:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105008 00000020:00000001:6.0:1496184947.960663:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.960665:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.960668:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.960670:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.960671:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.960673:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779800. 02000000:00000001:6.0:1496184947.960674:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960675:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.960677:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.960679:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.960681:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.960682:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.960684:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.960686:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.960688:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.960690:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.960691:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.960692:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.960693:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.960693:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.960694:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.960696:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.960697:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.960698:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.960699:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.960701:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.960702:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.960705:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.960706:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.960707:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.960709:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.960710:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.960712:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.960713:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.960714:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.960715:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.960717:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.960718:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.960719:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.960720:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.960721:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.960723:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.960725:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.960725:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.960726:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.960728:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.960729:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.960730:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.960731:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.960732:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.960734:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.960735:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960737:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.960739:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.960741:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.960743:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.960744:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.960746:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.960747:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960748:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.960749:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.960750:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.960752:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.960753:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.960754:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960755:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.960756:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.960757:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.960758:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.960760:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.960760:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.960762:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.960763:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.960764:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.960765:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.960767:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.960768:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.960769:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.960771:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.960772:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.960773:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.960774:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.960775:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.960776:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.960777:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.960778:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105008 00010000:00000001:6.0:1496184947.960780:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.960782:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7c50 x1568861104105008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.960787:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.960788:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.960789:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.960791:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.960793:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.960794:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.960795:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.960796:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.960797:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960798:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.960800:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.960802:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105008, offset 0 00000100:00000001:6.0:1496184947.960813:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.960814:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.960815:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.960816:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.960817:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.960818:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.960819:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.960820:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7c50 x1568861104105008/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.960826:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105008:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (271us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.960830:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43373 00000100:00000040:6.0:1496184947.960832:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.960833:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.960834:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.960836:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26380. 00000020:00000010:6.0:1496184947.960837:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.960839:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.960840:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.960841:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.960843:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.960846:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.960847:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184947.960910:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184947.960912:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184947.960913:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779800. 02000000:00000001:7.0:1496184947.960916:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184947.960917:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184947.961369:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184947.961371:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184947.961373:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184947.961379:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184947.961380:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184947.961382:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184947.961413:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.961416:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.961417:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.961418:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.961419:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.961420:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.961423:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.961425:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184947.961426:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.961427:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.961428:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.961430:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.961431:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.961433:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.961435:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.961435:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.961437:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.961439:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26a00. 00000020:00000010:6.0:1496184947.961441:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184947.961443:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184947.961444:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.961445:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184947.961446:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184947.961447:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.961448:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.961450:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.961451:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184947.961452:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184947.961453:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.961454:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184947.961455:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.961456:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.961457:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.961458:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184947.961458:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961459:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184947.961461:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961462:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961463:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.961464:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184947.961465:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.961466:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.961469:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184947.961475:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.961476:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.961478:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.961479:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.961480:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.961481:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.961483:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.961483:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.961484:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.961484:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00002000:00000001:6.0:1496184947.961486:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.961487:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.961489:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.961493:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.961495:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.961496:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.961497:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.961498:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.961499:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.961500:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.961502:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57628 00000100:00000040:6.0:1496184947.961504:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.961505:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671699536 : -131872037852080 : ffff881026072c50) 00000100:00000040:6.0:1496184947.961507:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.961513:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.961513:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.961515:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184947.961517:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184947.961518:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184947.961520:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184947.961521:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961522:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.961522:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184947.961524:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.961525:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.961526:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.961526:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.961527:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961528:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184947.961530:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.961531:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.961532:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77d800. 02000000:00000001:6.0:1496184947.961533:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.961534:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961536:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184947.961536:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184947.961538:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184947.961543:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184947.961545:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184947.961546:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.961548:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.961550:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184947.961551:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.961552:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.961553:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.961555:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.961556:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.961558:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.961559:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184947.961560:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184947.961561:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184947.961561:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184947.961563:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184947.961564:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184947.961565:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184947.961567:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184947.961589:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184947.961595:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961597:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184947.961599:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.961600:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.961602:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.961602:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.961604:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184947.961605:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c000. 00000100:00000010:6.0:1496184947.961606:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184947.961608:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.961612:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184947.961613:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184947.961640:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe640-0x592def0cfe640 00000100:00000001:6.0:1496184947.961642:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.963210:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.963212:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c000 00000100:00000001:3.0:1496184947.963218:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184947.963223:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.963223:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c000 00000100:00000001:3.0:1496184947.963228:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184947.963265:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184947.963274:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.963277:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184947.963280:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184947.963281:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184947.963284:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184947.963285:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184947.963318:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184947.963320:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184947.963322:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184947.963323:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184947.963324:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184947.963325:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.963327:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184947.963328:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184947.963329:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184947.963329:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184947.963330:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184947.963332:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184947.963334:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c000. 00000100:00000001:6.0:1496184947.963335:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184947.963336:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184947.963337:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77d800. 02000000:00000001:6.0:1496184947.963338:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184947.963340:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184947.963342:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184947.963344:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184947.963350:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184947.963351:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184947.963352:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.963352:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.963354:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.963356:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184947.963366:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1852us (1990us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184947.963371:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57628 00000100:00000040:6.0:1496184947.963372:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.963374:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184947.963375:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184947.963377:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184947.963383:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184947.963384:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184947.963385:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184947.963386:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184947.963387:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.963389:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184947.963390:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184947.963391:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184947.963392:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184947.963392:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00002000:00000001:6.0:1496184947.963393:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.963395:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.963396:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26a00. 00000020:00000010:6.0:1496184947.963398:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.963399:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.963401:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.963402:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.963404:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184947.963410:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.963411:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.991700:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.991703:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.991704:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184947.991706:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184947.991707:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184947.991708:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184947.991710:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184947.991712:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105040 00000020:00000001:6.0:1496184947.991714:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184947.991715:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184947.991716:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184947.991717:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184947.991719:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184947.991720:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184947.991722:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.991723:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184947.991725:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184947.991727:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26400. 00000020:00000010:6.0:1496184947.991728:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184947.991730:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184947.991731:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184947.991732:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184947.991734:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184947.991737:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6c50 x1568861104105040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.991742:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184947.991743:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991747:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184947.991749:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.991750:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184947.991751:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184947.991752:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184947.991753:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184947.991753:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184947.991756:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43374 00000100:00000040:6.0:1496184947.991757:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.991758:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676237904 : -131872033313712 : ffff8810264c6c50) 00000100:00000040:6.0:1496184947.991761:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6c50 x1568861104105040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184967 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184947.991766:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184947.991767:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184947.991769:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105040:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184947.991771:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105040 00000020:00000001:6.0:1496184947.991772:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184947.991774:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184947.991777:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184947.991780:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184947.991781:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184947.991782:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77d800. 02000000:00000001:6.0:1496184947.991783:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991784:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.991787:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184947.991788:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184947.991790:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184947.991792:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184947.991793:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184947.991795:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184947 last 1496184947 00010000:00000001:6.0:1496184947.991798:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184947.991799:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184947.991800:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184947.991801:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184947.991802:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184947.991803:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184947.991804:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184947.991805:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184947.991806:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184947.991807:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184947.991808:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184947.991809:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184947.991811:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184947.991814:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184947.991815:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184947.991816:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184947.991818:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184947.991819:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184947.991821:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184947.991822:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184947.991823:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184947.991824:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184947.991826:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184947.991827:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184947.991828:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.991829:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184947.991830:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184947.991832:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184947.991833:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184947.991834:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.991835:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184947.991837:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184947.991838:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.991839:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184947.991840:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184947.991841:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.991843:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.991844:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991845:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184947.991848:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184947.991850:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184947.991851:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184947.991853:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.991854:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.991855:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991856:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.991858:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.991859:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.991860:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184947.991862:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184947.991863:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991864:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184947.991865:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184947.991866:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184947.991867:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184947.991868:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184947.991869:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184947.991870:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184947.991872:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184947.991873:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184947.991874:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184947.991875:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.991877:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184947.991878:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184947.991879:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184947.991880:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184947.991881:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184947.991882:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184947.991883:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.991884:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184947.991885:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184947.991886:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105040 00010000:00000001:6.0:1496184947.991888:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184947.991904:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6c50 x1568861104105040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184947.991909:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184947.991910:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184947.991911:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=92 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184947.991914:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184947.991916:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184947.991917:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.991918:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184947.991927:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184947.991928:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991929:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184947.991930:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184947.991932:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105040, offset 0 00000100:00000001:6.0:1496184947.991943:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184947.991944:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184947.991946:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184947.991947:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184947.991948:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184947.991948:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184947.991949:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184947.991951:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6c50 x1568861104105040/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184967 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184947.991957:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105040:12345-10.37.248.124@o2ib1:8 Request procesed in 190us (303us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184947.991962:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43374 00000100:00000040:6.0:1496184947.991963:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184947.991965:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184947.991966:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184947.991967:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26400. 00000020:00000010:6.0:1496184947.991969:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184947.991970:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184947.991972:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184947.991973:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184947.991975:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184947.991978:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184947.991979:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184947.992537:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184947.992540:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184947.992542:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184947.992543:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184947.992553:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184947.992554:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184947.992556:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184947.992558:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184947.992560:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184947.992560:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184947.992562:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184947.992563:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184947.992564:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184947.992566:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184947.992568:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184947.992569:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184947.992571:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184947.992572:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3fa00. 00000020:00000010:4.0:1496184947.992574:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184947.992576:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184947.992578:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184947.992578:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184947.992579:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184947.992581:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184947.992582:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184947.992583:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184947.992584:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184947.992585:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184947.992586:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184947.992588:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184947.992590:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184947.992591:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184947.992592:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184947.992593:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184947.992594:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992595:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184947.992596:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992597:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992598:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184947.992599:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184947.992600:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184947.992601:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184947.992604:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184947.992611:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184947.992611:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184947.992613:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184947.992614:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184947.992615:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184947.992616:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184947.992618:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184947.992618:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184947.992619:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184947.992620:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00002000:00000001:4.0:1496184947.992621:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184947.992622:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184947.992624:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184947.992629:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184947.992631:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184947.992632:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184947.992633:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184947.992633:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184947.992634:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184947.992635:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184947.992638:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57629 00000100:00000040:4.0:1496184947.992640:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184947.992641:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671698512 : -131872037853104 : ffff881026072850) 00000100:00000040:4.0:1496184947.992643:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/0 e 0 to 0 dl 1496184990 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184947.992648:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184947.992649:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184947.992651:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184947.992653:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184947.992654:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184947.992656:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184947.992656:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992657:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184947.992658:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184947.992660:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184947.992661:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184947.992662:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184947.992662:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184947.992663:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992664:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184947.992666:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184947.992667:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184947.992668:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184947.992669:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184947.992670:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992672:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184947.992673:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184947.992674:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184947.992681:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184947.992683:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184947.992684:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184947.992687:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184947.992689:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184947.992691:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184947.992691:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184947.992693:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184947.992695:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184947.992696:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184947.992698:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184947.992699:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184947.992700:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184947.992700:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184947.992701:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184947.992703:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184947.992704:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184947.992705:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184947.992707:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184947.992729:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184947.992735:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992736:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184947.992739:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184947.992741:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184947.992742:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184947.992743:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184947.992744:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184947.992745:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184947.992747:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184947.992748:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184947.992753:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184947.992754:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184947.992780:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe660-0x592def0cfe660 00000100:00000001:4.0:1496184947.992782:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184947.994451:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184947.994454:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.005606:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.005611:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.005612:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.016609:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.016776:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.016783:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.016785:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.016791:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.016792:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.016794:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.016796:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.016829:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.016831:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.016833:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.016834:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.016834:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.016836:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.016837:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.016838:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.016839:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.016840:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.016841:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.016843:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.016844:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.016846:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.016847:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.016848:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.016849:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.016851:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.016852:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.016855:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.016860:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.016862:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.016862:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.016863:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.016864:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.016866:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.016872:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24222us (24372us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.016877:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57629 00000100:00000040:4.0:1496184948.016879:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.016880:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.016881:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.016884:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:735/0 lens 608/448 e 0 to 0 dl 1496184990 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.016901:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.016902:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.016904:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.016905:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.016907:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.016908:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.016910:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.016910:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.016911:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.016912:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00002000:00000001:4.0:1496184948.016913:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.016914:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.016916:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3fa00. 00000020:00000010:4.0:1496184948.016917:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.016919:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.016920:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.016922:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.016924:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.016929:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.016930:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.027332:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.027335:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.027337:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.027338:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.027339:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.027340:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.027342:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.027344:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105072 00000020:00000001:6.0:1496184948.027346:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.027346:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.027348:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.027349:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.027350:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.027352:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.027354:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.027355:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.027356:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7de00. 00000020:00000010:6.0:1496184948.027359:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26d00. 00000020:00000010:6.0:1496184948.027360:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.027362:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.027363:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.027364:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.027366:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.027369:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7050 x1568861104105072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.027374:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.027375:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027380:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.027382:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.027383:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.027384:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.027384:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.027385:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.027386:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.027389:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43375 00000100:00000040:6.0:1496184948.027390:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.027392:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676238928 : -131872033312688 : ffff8810264c7050) 00000100:00000040:6.0:1496184948.027394:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7050 x1568861104105072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.027399:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.027400:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.027402:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105072:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.027404:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105072 00000020:00000001:6.0:1496184948.027405:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.027407:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.027410:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.027412:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.027413:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.027415:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a800. 02000000:00000001:6.0:1496184948.027416:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027417:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.027420:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.027421:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.027423:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.027424:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.027426:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.027428:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.027430:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.027432:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.027432:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.027434:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.027434:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.027435:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.027437:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.027438:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.027439:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.027440:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.027441:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.027443:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.027444:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.027447:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.027448:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.027449:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.027451:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.027452:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.027454:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.027455:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.027456:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.027457:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.027459:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.027460:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.027462:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.027463:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.027464:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.027465:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.027467:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.027468:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.027469:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.027470:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.027471:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.027472:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.027473:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.027474:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.027476:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.027477:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027478:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.027481:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.027483:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.027485:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.027486:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.027487:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.027489:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027490:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.027491:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.027492:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.027494:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.027495:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.027496:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027497:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.027498:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.027499:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.027500:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.027502:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.027502:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.027503:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.027505:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.027506:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.027507:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.027509:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.027510:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.027511:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.027513:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.027514:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.027515:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.027515:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.027516:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.027517:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.027518:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.027520:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105072 00010000:00000001:6.0:1496184948.027521:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.027523:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7050 x1568861104105072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.027529:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.027529:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.027531:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.027533:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.027534:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.027536:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.027537:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.027538:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.027539:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027540:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.027542:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.027544:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105072, offset 0 00000100:00000001:6.0:1496184948.027554:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.027555:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.027557:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.027558:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.027559:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.027560:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.027561:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.027562:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7050 x1568861104105072/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.027568:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105072:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (276us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.027572:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43375 00000100:00000040:6.0:1496184948.027574:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.027575:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.027576:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.027577:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26d00. 00000020:00000010:6.0:1496184948.027579:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.027580:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7de00. 00000020:00000040:6.0:1496184948.027582:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.027583:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.027585:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.027588:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.027589:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.027641:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.027644:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.027645:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77a800. 02000000:00000001:7.0:1496184948.027648:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.027649:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.028124:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.028125:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.028128:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.028141:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.028142:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.028145:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:4.0:1496184948.028175:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.028177:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.028179:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.028180:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.028181:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.028182:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.028184:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.028188:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.028190:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.028191:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.028192:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.028193:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.028195:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.028196:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.028199:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.028199:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.028201:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.028203:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3f700. 00000020:00000010:4.0:1496184948.028204:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.028207:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.028209:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.028209:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.028210:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.028212:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.028213:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.028214:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.028215:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.028216:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.028217:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.028219:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.028219:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.028221:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.028221:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.028222:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.028223:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028224:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.028225:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028226:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028227:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.028229:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.028230:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.028231:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.028234:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.028247:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.028248:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.028250:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.028251:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.028252:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.028253:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.028254:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.028255:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.028256:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.028256:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00002000:00000001:4.0:1496184948.028258:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.028259:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.028261:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.028265:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.028267:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.028268:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.028269:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.028270:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.028271:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.028272:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.028274:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57630 00000100:00000040:4.0:1496184948.028276:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.028277:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671696464 : -131872037855152 : ffff881026072050) 00000100:00000040:4.0:1496184948.028279:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.028284:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.028285:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.028287:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.028289:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.028290:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.028292:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.028293:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028294:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.028295:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.028296:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.028297:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.028298:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.028299:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.028300:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028301:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.028303:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.028304:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.028305:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.028306:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.028307:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028309:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.028309:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.028311:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.028316:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.028318:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.028319:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.028321:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.028323:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.028324:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.028325:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.028326:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.028328:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.028329:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.028331:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.028332:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.028333:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.028333:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.028334:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.028336:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.028337:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.028338:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.028341:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.028362:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.028368:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028369:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.028372:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.028373:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.028374:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.028375:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.028376:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.028378:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.028379:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.028380:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.028385:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.028386:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.028412:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe680-0x592def0cfe680 00000100:00000001:4.0:1496184948.028414:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.030006:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.030020:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.041061:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.041067:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.041068:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.052112:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.052179:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.052199:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1496184948.052200:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000020:00000001:4.0:1496184948.052201:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000200:1.0:1496184948.052202:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:4.0:1496184948.052205:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.052205:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000040:1.0:1496184948.052205:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d850 x1568861104105104 msgsize 520 00002000:00000001:4.0:1496184948.052208:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.052209:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184948.052212:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.052213:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.052217:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.052223:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.052225:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.052229:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184948.052232:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.052234:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184948.052236:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184948.052240:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.052243:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184948.052244:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105104 00000020:00000002:4.0:1496184948.052245:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000020:00000001:3.0:1496184948.052246:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:4.0:1496184948.052247:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.052248:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00000020:00000040:3.0:1496184948.052248:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00000001:4.0:1496184948.052249:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.052250:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1496184948.052250:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.052251:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.052253:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000040:3.0:1496184948.052253:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184948.052255:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.052256:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.052256:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000001:3.0:1496184948.052257:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000040:4.0:1496184948.052258:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000010:4.0:1496184948.052259:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.052261:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:3.0:1496184948.052261:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.052262:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.052263:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.052265:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 00000100:00000001:3.0:1496184948.052265:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 02000000:00000001:4.0:1496184948.052266:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.052268:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000010:3.0:1496184948.052268:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88085876f400. 00000020:00000001:4.0:1496184948.052269:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.052272:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184948.052272:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132fc00. 00000020:00000010:3.0:1496184948.052275:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2cc0. 00010000:00000040:4.0:1496184948.052277:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000001:3.0:1496184948.052277:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.052278:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00010000:00000001:4.0:1496184948.052279:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.052279:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.052280:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052280:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.052281:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.052283:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.052285:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:4.0:1496184948.052289:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24003us (24159us total) trans 0 rc -110/-999 00000100:00000200:3.0:1496184948.052290:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8d850 x1568861104105104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:4.0:1496184948.052294:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57630 00000100:00000040:4.0:1496184948.052296:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.052296:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184948.052297:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.052298:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.052299:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.052301:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.052301:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184948.052305:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.052306:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00002000:00000001:4.0:1496184948.052307:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.052307:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184948.052308:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.052308:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00010000:00000001:4.0:1496184948.052309:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000100:00000001:3.0:1496184948.052310:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000020:00000001:4.0:1496184948.052311:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:3.0:1496184948.052311:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000020:00000001:4.0:1496184948.052312:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.052314:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.052315:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00000100:00100000:3.0:1496184948.052315:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6094 00010000:00000001:4.0:1496184948.052316:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.052317:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.052317:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00000100:00000040:3.0:1496184948.052317:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00000001:4.0:1496184948.052318:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052318:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736402512 : -131906973149104 : ffff880803b8d850) 00000100:00000001:4.0:1496184948.052320:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000040:3.0:1496184948.052321:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d850 x1568861104105104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000010:4.0:1496184948.052322:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3f700. 00000020:00000010:4.0:1496184948.052323:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.052325:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.052326:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:3.0:1496184948.052327:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.052328:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184948.052328:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00000200:4.0:1496184948.052330:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00100000:3.0:1496184948.052330:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105104:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184948.052333:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105104 00004000:00000001:4.0:1496184948.052335:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000020:00000001:3.0:1496184948.052335:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00004000:00000001:4.0:1496184948.052337:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00000001:3.0:1496184948.052340:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184948.052347:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184948.052352:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184948.052354:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184948.052356:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23ff800. 02000000:00000001:3.0:1496184948.052359:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052361:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.052367:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184948.052369:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184948.052372:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.052374:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184948.052376:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184948.052378:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.052389:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.052392:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.052394:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.052396:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.052399:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.052400:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.052402:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.052404:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.052406:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.052407:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.052409:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.052410:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.052412:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.052417:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.052418:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.052420:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.052422:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.052424:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.052426:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.052428:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.052429:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.052431:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.052435:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.052436:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.052438:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.052440:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.052441:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.052443:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.052445:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.052446:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.052447:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.052449:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.052451:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.052452:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.052453:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.052456:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.052458:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.052459:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052461:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.052465:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.052467:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.052469:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.052472:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.052473:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.052474:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052476:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.052477:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.052478:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.052480:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.052483:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.052484:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052486:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.052489:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.052490:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.052492:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.052494:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.052495:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.052497:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.052499:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.052500:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.052502:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.052504:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.052507:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.052509:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.052511:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.052512:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.052513:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.052514:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.052516:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.052517:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.052520:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.052522:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105104 00010000:00000001:3.0:1496184948.052532:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.052534:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8d850 x1568861104105104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.052541:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.052543:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.052544:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.052547:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.052549:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.052550:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.052552:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.052553:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.052555:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052556:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.052558:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.052562:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105104, offset 0 00000100:00000001:3.0:1496184948.052578:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.052579:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.052581:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.052582:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.052583:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.052584:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.052585:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.052587:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d850 x1568861104105104/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.052593:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105104:12345-10.37.248.124@o2ib1:8 Request procesed in 265us (386us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.052598:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6094 00000100:00000040:3.0:1496184948.052600:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.052602:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.052602:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.052605:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132fc00. 00000020:00000010:3.0:1496184948.052608:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2cc0. 00000020:00000010:3.0:1496184948.052616:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88085876f400. 00000020:00000040:3.0:1496184948.052618:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.052620:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.052623:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.052629:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.052630:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.053386:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.053389:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.053390:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.053392:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.053393:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.053394:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.053397:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.053398:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.053400:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.053401:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.053402:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.053404:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.053405:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.053407:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.053409:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.053410:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.053412:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.053414:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3fc00. 00000020:00000010:4.0:1496184948.053418:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.053420:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.053422:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.053423:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.053424:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.053425:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.053426:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.053428:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.053429:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.053430:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.053431:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.053432:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.053433:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.053434:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.053435:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.053436:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.053436:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053438:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.053439:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053440:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053441:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.053442:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.053443:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.053444:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.053447:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.053454:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.053454:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.053456:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.053457:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.053458:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.053459:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.053461:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.053461:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.053462:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.053463:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00002000:00000001:4.0:1496184948.053465:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.053466:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.053468:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.053473:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.053475:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.053476:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.053477:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.053478:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.053479:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.053480:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.053482:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57631 00000100:00000040:4.0:1496184948.053484:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.053485:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671693392 : -131872037858224 : ffff881026071450) 00000100:00000040:4.0:1496184948.053488:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.053493:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.053494:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.053496:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.053498:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.053499:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.053501:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.053502:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053503:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.053503:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.053505:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.053506:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.053507:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.053508:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.053509:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053510:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.053512:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.053512:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.053514:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.053516:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.053517:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053519:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.053519:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.053521:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.053525:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.053528:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.053529:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.053531:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.053533:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.053534:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.053535:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.053536:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.053538:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.053539:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.053541:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.053543:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.053544:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.053544:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.053545:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.053547:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.053548:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.053549:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.053551:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.053573:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.053578:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053580:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.053582:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.053584:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.053585:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.053586:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.053587:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.053588:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.053590:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.053592:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.053596:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.053597:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.053624:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe6a0-0x592def0cfe6a0 00000100:00000001:4.0:1496184948.053627:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.054951:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.054954:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.065957:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.065962:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.065962:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.076944:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.077005:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.077025:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.077027:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.077031:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.077032:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.077034:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.077036:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.077069:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.077071:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.077073:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.077074:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.077074:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.077076:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.077077:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.077078:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.077079:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.077080:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.077081:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.077083:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.077084:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.077086:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.077087:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.077088:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.077089:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.077091:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.077092:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.077097:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.077102:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.077104:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.077105:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.077105:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.077107:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.077108:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.077114:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23620us (23767us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.077119:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57631 00000100:00000040:4.0:1496184948.077121:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.077122:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.077123:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.077126:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.077132:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.077132:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.077134:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.077135:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.077136:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.077138:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.077139:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.077140:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.077141:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.077141:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00002000:00000001:4.0:1496184948.077142:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.077144:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.077145:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3fc00. 00000020:00000010:4.0:1496184948.077147:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.077149:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.077151:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.077152:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.077154:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.077160:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.077161:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.091246:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.091249:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.091251:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.091252:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.091253:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.091254:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.091257:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.091258:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105136 00000020:00000001:6.0:1496184948.091260:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.091261:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.091262:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.091263:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.091265:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.091266:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.091268:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.091269:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.091271:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.091273:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26980. 00000020:00000010:6.0:1496184948.091274:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.091276:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.091277:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.091278:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.091280:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.091283:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7850 x1568861104105136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.091288:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.091289:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091294:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.091296:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.091296:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.091297:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.091298:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.091299:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.091300:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.091302:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43376 00000100:00000040:6.0:1496184948.091304:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.091305:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676240976 : -131872033310640 : ffff8810264c7850) 00000100:00000040:6.0:1496184948.091307:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7850 x1568861104105136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.091313:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.091313:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.091315:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105136:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.091317:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105136 00000020:00000001:6.0:1496184948.091318:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.091320:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.091324:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.091326:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.091327:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.091328:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.091330:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091331:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.091333:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.091335:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.091337:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.091338:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.091340:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.091342:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.091345:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.091346:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.091347:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.091348:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.091350:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.091351:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.091352:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.091353:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.091355:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.091356:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.091357:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.091359:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.091360:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.091363:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.091364:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.091365:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.091367:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.091368:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.091370:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.091371:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.091372:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.091373:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.091375:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.091377:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.091378:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.091379:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.091380:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.091382:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.091384:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.091385:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.091386:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.091387:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.091389:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.091390:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.091390:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.091392:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.091394:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.091395:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091396:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.091399:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.091401:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.091403:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.091405:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.091406:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.091407:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091408:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.091409:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.091411:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.091412:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.091414:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.091415:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091416:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.091417:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.091418:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.091419:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.091421:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.091421:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.091423:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.091424:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.091426:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.091427:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.091428:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.091430:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.091431:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.091432:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.091433:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.091434:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.091435:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.091436:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.091437:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.091438:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.091440:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105136 00010000:00000001:6.0:1496184948.091441:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.091443:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7850 x1568861104105136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.091448:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.091449:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.091450:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.091453:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.091454:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.091456:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.091457:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.091458:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.091459:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091460:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.091462:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.091464:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105136, offset 0 00000100:00000001:6.0:1496184948.091475:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.091476:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.091477:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.091478:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.091479:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.091480:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.091481:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.091483:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7850 x1568861104105136/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.091488:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105136:12345-10.37.248.124@o2ib1:8 Request procesed in 175us (283us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.091493:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43376 00000100:00000040:6.0:1496184948.091494:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.091496:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.091496:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.091498:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26980. 00000020:00000010:6.0:1496184948.091500:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.091501:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.091503:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.091504:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.091506:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.091509:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.091510:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.092026:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.092029:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.092031:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.092032:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.092033:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.092034:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.092037:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.092039:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.092040:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.092041:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.092042:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.092043:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.092045:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.092046:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.092049:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.092049:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.092051:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.092053:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3fd80. 00000020:00000010:4.0:1496184948.092055:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.092057:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.092058:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.092059:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.092060:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.092061:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.092062:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.092064:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.092065:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.092066:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.092067:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.092068:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.092071:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.092072:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.092073:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.092074:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.092075:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092076:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.092077:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092078:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092079:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.092080:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.092081:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.092083:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.092085:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.092092:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.092093:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.092094:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.092095:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.092096:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.092097:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.092099:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.092100:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.092100:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.092101:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00002000:00000001:4.0:1496184948.092102:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.092103:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.092106:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.092110:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.092112:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.092113:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.092114:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.092115:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.092116:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.092117:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.092119:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57632 00000100:00000040:4.0:1496184948.092122:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.092123:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671272528 : -131872038279088 : ffff88102600a850) 00000100:00000040:4.0:1496184948.092125:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.092130:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.092131:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.092133:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.092135:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.092136:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.092138:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.092138:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092140:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.092140:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.092142:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.092143:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.092144:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.092144:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.092145:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092146:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.092148:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.092149:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.092150:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.092151:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.092152:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092154:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.092155:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.092156:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.092161:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.092163:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.092164:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.092166:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.092168:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.092170:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.092171:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.092172:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.092174:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.092175:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.092177:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.092178:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.092179:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.092180:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.092181:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.092183:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.092183:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.092184:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.092187:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.092208:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.092214:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092215:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.092218:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.092219:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.092220:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.092221:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.092222:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.092223:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.092225:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.092226:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.092231:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.092232:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.092258:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe6c0-0x592def0cfe6c0 00000100:00000001:4.0:1496184948.092260:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.093782:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.093784:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.104810:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.104815:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.104816:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.115802:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.115865:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.115884:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.115886:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.115902:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.115903:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.115905:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.115907:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.115940:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.115942:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.115944:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.115945:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.115946:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.115947:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.115948:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.115950:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.115950:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.115951:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.115952:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.115954:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.115955:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.115957:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.115958:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.115959:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.115960:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.115962:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.115963:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.115966:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.115971:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.115973:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.115973:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.115974:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.115975:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.115977:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.115983:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23851us (23998us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.115990:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57632 00000100:00000040:4.0:1496184948.115991:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.115993:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.115994:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.115996:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.116002:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.116002:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.116004:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.116005:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.116006:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.116008:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.116009:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.116010:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.116011:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.116011:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00002000:00000001:4.0:1496184948.116013:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.116014:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.116015:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3fd80. 00000020:00000010:4.0:1496184948.116017:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.116019:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.116020:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.116021:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.116024:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.116029:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.116030:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.127056:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.127057:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.127060:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6850 x1568861104105168 msgsize 520 00000100:00100000:7.0:1496184948.127065:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.127073:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.127075:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.127111:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.127113:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.127115:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.127116:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.127117:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.127118:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.127121:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.127123:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105168 00000020:00000001:6.0:1496184948.127124:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.127125:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.127126:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.127127:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.127129:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.127131:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.127133:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.127133:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.127135:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7fe00. 00000020:00000010:6.0:1496184948.127137:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26e80. 00000020:00000010:6.0:1496184948.127138:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.127140:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.127141:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.127142:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.127144:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.127147:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6850 x1568861104105168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.127152:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.127153:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127158:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.127160:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.127161:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.127162:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.127162:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.127163:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.127164:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.127166:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43377 00000100:00000040:6.0:1496184948.127168:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.127169:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676236880 : -131872033314736 : ffff8810264c6850) 00000100:00000040:6.0:1496184948.127171:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6850 x1568861104105168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.127177:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.127177:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.127179:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105168:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.127181:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105168 00000020:00000001:6.0:1496184948.127182:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.127184:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.127187:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.127190:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.127191:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.127192:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77e000. 02000000:00000001:6.0:1496184948.127193:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127195:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.127197:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.127199:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.127200:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.127202:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.127203:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.127205:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.127208:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.127209:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.127210:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.127211:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.127212:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.127213:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.127214:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.127215:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.127217:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.127218:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.127219:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.127220:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.127221:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.127224:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.127225:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.127226:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.127228:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.127229:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.127231:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.127232:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.127233:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.127234:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.127236:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.127237:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.127239:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.127240:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.127241:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.127242:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.127244:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.127245:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.127246:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.127247:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.127248:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.127249:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.127250:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.127252:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.127253:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.127255:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127256:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.127258:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.127260:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.127262:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.127263:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.127265:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.127266:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127267:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.127268:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.127269:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.127271:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.127272:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.127273:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127274:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.127275:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.127276:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.127277:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.127278:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.127279:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.127280:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.127282:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.127283:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.127284:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.127286:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.127287:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.127288:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.127290:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.127290:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.127292:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.127292:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.127293:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.127294:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.127295:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.127297:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105168 00010000:00000001:6.0:1496184948.127298:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.127300:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6850 x1568861104105168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.127305:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.127306:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.127307:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.127309:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.127311:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.127312:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.127314:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.127315:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.127316:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127317:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.127318:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.127320:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105168, offset 0 00000100:00000001:6.0:1496184948.127331:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.127332:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.127333:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.127334:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.127336:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.127336:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.127337:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.127339:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6850 x1568861104105168/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.127345:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105168:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (283us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.127349:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43377 00000100:00000040:6.0:1496184948.127351:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.127352:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.127353:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.127354:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26e80. 00000020:00000010:6.0:1496184948.127356:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.127357:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7fe00. 00000020:00000040:6.0:1496184948.127359:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.127360:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.127362:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.127365:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.127366:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.127433:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.127436:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.127437:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77e000. 02000000:00000001:7.0:1496184948.127440:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.127441:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.127973:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.127975:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.127983:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.127988:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.127989:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.127991:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:4.0:1496184948.128024:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.128026:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.128028:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.128029:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.128030:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.128031:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.128033:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.128037:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.128039:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.128040:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.128041:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.128042:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.128043:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.128045:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.128047:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.128048:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.128049:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.128051:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3fb00. 00000020:00000010:4.0:1496184948.128053:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.128055:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.128057:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.128057:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.128058:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.128060:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.128061:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.128062:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.128063:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.128064:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.128065:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.128067:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.128067:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.128068:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.128069:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.128070:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.128071:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128072:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.128073:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128074:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128075:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.128076:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.128077:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.128079:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.128081:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.128088:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.128089:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.128091:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.128092:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.128093:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.128094:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.128096:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.128096:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.128097:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.128098:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00002000:00000001:4.0:1496184948.128099:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.128100:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.128102:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.128107:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.128109:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.128109:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.128110:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.128111:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.128112:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.128113:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.128115:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57633 00000100:00000040:4.0:1496184948.128117:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.128118:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671694416 : -131872037857200 : ffff881026071850) 00000100:00000040:4.0:1496184948.128121:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.128126:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.128126:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.128128:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.128130:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.128131:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.128133:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.128134:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128135:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.128136:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.128138:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.128139:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.128140:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.128140:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.128141:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128142:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.128144:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.128145:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.128146:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.128147:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.128149:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128150:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.128151:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.128153:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.128157:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.128159:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.128160:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.128162:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.128164:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.128165:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.128166:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.128167:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.128169:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.128170:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.128172:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.128173:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.128174:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.128175:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.128176:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.128177:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.128179:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.128180:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.128182:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.128205:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.128211:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128212:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.128215:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.128216:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.128217:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.128218:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.128219:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.128220:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.128222:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.128223:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.128228:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.128229:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.128254:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe6e0-0x592def0cfe6e0 00000100:00000001:4.0:1496184948.128256:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.130047:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.130052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.141107:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.141112:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.141113:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.152108:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.152182:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.152189:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.152191:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.152195:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.152196:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.152198:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.152200:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.152233:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.152235:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.152237:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.152238:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.152238:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.152240:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.152241:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.152242:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.152245:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.152246:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.152247:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.152249:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.152250:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.152252:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.152252:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.152254:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.152255:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.152257:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.152258:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.152261:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.152266:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.152267:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.152268:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.152269:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.152270:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.152272:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.152278:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24151us (24293us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.152283:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57633 00000100:00000040:4.0:1496184948.152284:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.152286:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.152287:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.152289:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.152295:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.152296:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.152297:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.152299:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.152300:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.152301:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.152303:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.152303:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.152304:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.152305:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00002000:00000001:4.0:1496184948.152306:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.152307:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.152309:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3fb00. 00000020:00000010:4.0:1496184948.152310:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.152312:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.152313:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.152315:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.152317:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.152322:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.152323:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.163001:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.163004:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.163006:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.163007:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.163008:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.163009:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.163011:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.163013:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105200 00000020:00000001:6.0:1496184948.163015:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.163015:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.163017:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.163018:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.163019:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.163021:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.163023:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.163024:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.163025:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7f600. 00000020:00000010:6.0:1496184948.163027:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26f00. 00000020:00000010:6.0:1496184948.163029:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.163030:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.163031:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.163032:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.163034:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.163037:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6050 x1568861104105200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.163043:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.163044:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163048:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.163050:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.163051:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.163052:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.163053:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.163053:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.163054:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.163057:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43378 00000100:00000040:6.0:1496184948.163058:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.163059:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676234832 : -131872033316784 : ffff8810264c6050) 00000100:00000040:6.0:1496184948.163062:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6050 x1568861104105200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.163067:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.163067:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.163069:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105200:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.163071:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105200 00000020:00000001:6.0:1496184948.163072:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.163074:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.163077:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.163080:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.163081:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.163082:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77b000. 02000000:00000001:6.0:1496184948.163083:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163085:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.163087:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.163089:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.163090:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.163092:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.163093:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.163095:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.163098:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.163099:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.163100:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.163101:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.163102:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.163103:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.163104:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.163105:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.163107:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.163107:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.163109:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.163110:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.163111:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.163114:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.163115:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.163116:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.163118:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.163119:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.163121:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.163122:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.163123:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.163124:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.163126:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.163127:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.163128:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.163130:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.163130:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.163132:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.163133:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.163134:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.163135:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.163137:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.163138:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.163139:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.163140:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.163142:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.163143:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.163144:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163146:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.163148:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.163150:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.163152:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.163153:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.163155:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.163156:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163157:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.163158:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.163159:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.163161:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.163162:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.163163:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163164:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.163165:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.163166:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.163167:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.163169:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.163169:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.163171:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.163172:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.163174:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.163175:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.163176:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.163177:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.163178:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.163180:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.163181:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.163182:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.163183:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.163184:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.163185:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.163186:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.163187:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105200 00010000:00000001:6.0:1496184948.163189:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.163191:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6050 x1568861104105200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.163196:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.163197:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.163198:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.163200:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.163202:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.163203:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.163204:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.163206:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.163206:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163207:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.163209:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.163211:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105200, offset 0 00000100:00000001:6.0:1496184948.163222:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.163223:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.163224:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.163225:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.163226:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.163227:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.163228:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.163230:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6050 x1568861104105200/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.163235:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105200:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (271us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.163240:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43378 00000100:00000040:6.0:1496184948.163242:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.163243:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.163244:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.163245:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26f00. 00000020:00000010:6.0:1496184948.163247:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.163248:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7f600. 00000020:00000040:6.0:1496184948.163250:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.163251:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.163253:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.163256:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.163257:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.163773:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.163776:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.163777:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.163778:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.163779:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.163780:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.163783:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.163785:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.163786:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.163787:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.163788:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.163789:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.163791:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.163792:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.163794:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.163795:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.163797:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.163799:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3fe00. 00000020:00000010:4.0:1496184948.163802:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.163804:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.163806:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.163806:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.163807:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.163809:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.163810:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.163811:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.163812:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.163813:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.163814:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.163815:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.163816:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.163817:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.163818:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.163819:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.163820:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163821:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.163822:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163823:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163824:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.163825:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.163826:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.163827:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.163830:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.163837:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.163837:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.163839:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.163840:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.163841:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.163842:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.163844:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.163844:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.163845:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.163846:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00002000:00000001:4.0:1496184948.163847:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.163848:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.163850:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.163855:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.163857:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.163858:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.163859:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.163860:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.163861:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.163862:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.163864:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57634 00000100:00000040:4.0:1496184948.163865:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.163867:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671691344 : -131872037860272 : ffff881026070c50) 00000100:00000040:4.0:1496184948.163869:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.163874:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.163875:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.163877:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.163879:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.163880:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.163881:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.163882:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163883:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.163884:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.163885:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.163886:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.163887:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.163888:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.163889:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163902:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.163904:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.163905:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.163906:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.163908:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.163909:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163911:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.163911:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.163913:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.163918:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.163920:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.163921:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.163923:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.163925:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.163926:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.163927:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.163928:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.163930:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.163931:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.163933:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.163934:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.163935:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.163936:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.163936:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.163938:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.163939:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.163940:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.163942:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.163964:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.163970:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163971:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.163974:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.163975:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.163976:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.163977:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.163978:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.163979:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.163981:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.163983:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.163987:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.163988:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.164014:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe700-0x592def0cfe700 00000100:00000001:4.0:1496184948.164016:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.165410:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.165413:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.176437:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.176442:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.176443:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.187429:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.187492:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.187511:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.187513:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.187517:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.187518:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.187520:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.187522:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.187555:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.187557:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.187559:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.187560:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.187561:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.187562:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.187563:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.187565:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.187565:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.187566:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.187567:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.187569:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.187570:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.187572:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.187573:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.187574:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.187575:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.187577:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.187578:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.187583:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.187588:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.187590:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.187590:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.187591:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.187592:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.187594:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.187600:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23724us (23866us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.187605:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57634 00000100:00000040:4.0:1496184948.187606:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.187608:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.187609:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.187611:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.187617:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.187617:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.187619:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.187620:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.187621:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.187623:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.187624:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.187625:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.187625:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.187626:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00002000:00000001:4.0:1496184948.187627:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.187629:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.187630:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3fe00. 00000020:00000010:4.0:1496184948.187632:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.187634:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.187635:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.187637:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.187639:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.187644:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.187645:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.198166:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.198168:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.198170:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5c50 x1568861104105232 msgsize 520 00000100:00100000:7.0:1496184948.198184:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.198185:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.198187:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.198218:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.198221:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.198222:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.198224:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.198225:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.198226:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.198228:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.198230:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105232 00000020:00000001:6.0:1496184948.198232:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.198232:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.198233:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.198235:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.198236:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.198238:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.198240:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.198241:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.198242:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7c200. 00000020:00000010:6.0:1496184948.198245:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26880. 00000020:00000010:6.0:1496184948.198246:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.198248:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.198248:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.198249:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.198252:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.198255:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c5c50 x1568861104105232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.198264:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.198264:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198269:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.198271:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.198272:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.198273:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.198273:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.198274:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.198275:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.198277:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43379 00000100:00000040:6.0:1496184948.198279:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.198280:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676233808 : -131872033317808 : ffff8810264c5c50) 00000100:00000040:6.0:1496184948.198282:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5c50 x1568861104105232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.198288:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.198288:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.198290:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105232:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.198292:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105232 00000020:00000001:6.0:1496184948.198293:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.198295:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.198298:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.198301:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.198302:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.198303:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778400. 02000000:00000001:6.0:1496184948.198304:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198305:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.198308:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.198309:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.198311:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.198313:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.198314:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.198316:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.198319:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.198320:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.198321:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.198322:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.198323:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.198324:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.198325:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.198326:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.198328:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.198328:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.198329:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.198331:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.198332:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.198335:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.198336:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.198337:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.198339:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.198340:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.198342:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.198343:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.198344:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.198345:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.198347:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.198348:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.198349:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.198350:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.198351:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.198353:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.198354:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.198355:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.198356:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.198358:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.198359:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.198360:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.198361:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.198362:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.198364:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.198365:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198366:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.198369:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.198371:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.198372:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.198374:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.198375:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.198376:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198377:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.198379:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.198380:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.198381:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.198383:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.198384:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198385:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.198386:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.198387:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.198388:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.198389:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.198390:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.198391:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.198393:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.198394:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.198395:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.198396:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.198398:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.198399:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.198401:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.198401:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.198402:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.198403:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.198404:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.198405:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.198406:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.198408:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105232 00010000:00000001:6.0:1496184948.198409:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.198411:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c5c50 x1568861104105232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.198416:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.198417:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.198418:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.198420:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.198422:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.198423:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.198425:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.198426:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.198427:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198428:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.198429:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.198431:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105232, offset 0 00000100:00000001:6.0:1496184948.198442:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.198443:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.198445:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.198446:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.198447:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.198448:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.198449:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.198451:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5c50 x1568861104105232/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.198456:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105232:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (284us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.198461:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43379 00000100:00000040:6.0:1496184948.198462:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.198463:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.198464:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.198466:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26880. 00000020:00000010:6.0:1496184948.198467:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.198469:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7c200. 00000020:00000040:6.0:1496184948.198470:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.198472:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.198473:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.198477:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.198477:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.198986:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.198988:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.198990:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.198991:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.198992:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.198993:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.198995:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.198997:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.198999:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.198999:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.199001:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.199002:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.199003:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.199005:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.199007:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.199008:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.199009:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.199011:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3f900. 00000020:00000010:4.0:1496184948.199013:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.199015:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.199016:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.199017:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.199018:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.199019:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.199020:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.199021:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.199022:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.199024:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.199024:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.199026:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.199029:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.199030:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.199030:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.199031:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.199032:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199033:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.199034:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199035:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199036:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.199038:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.199039:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.199040:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.199043:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.199049:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.199050:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.199052:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.199052:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.199053:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.199055:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.199056:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.199057:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.199057:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.199058:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00002000:00000001:4.0:1496184948.199059:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.199061:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.199063:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.199067:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.199069:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.199070:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.199071:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.199072:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.199073:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.199074:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.199076:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57635 00000100:00000040:4.0:1496184948.199078:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.199079:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671692368 : -131872037859248 : ffff881026071050) 00000100:00000040:4.0:1496184948.199082:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.199087:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.199088:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.199089:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.199091:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.199092:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.199094:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.199095:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199096:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.199097:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.199098:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.199099:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.199100:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.199101:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.199102:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199103:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.199104:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.199105:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.199107:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.199108:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.199109:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199110:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.199111:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.199113:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.199117:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.199120:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.199121:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.199123:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.199125:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.199126:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.199127:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.199129:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.199130:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.199131:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.199133:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.199135:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.199135:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.199136:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.199137:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.199139:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.199139:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.199140:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.199143:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.199162:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.199168:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199169:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.199172:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.199173:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.199174:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.199175:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.199176:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.199177:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.199179:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.199180:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.199184:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.199185:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.199212:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe720-0x592def0cfe720 00000100:00000001:4.0:1496184948.199214:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.200780:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.200782:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.211819:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.211824:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.211825:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.222815:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.222887:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.222905:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.222907:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.222919:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.222920:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.222922:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.222924:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.222957:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.222959:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.222961:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.222962:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.222963:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.222964:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.222965:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.222967:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.222967:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.222968:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.222969:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.222971:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.222972:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.222974:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.222975:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.222976:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.222977:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.222979:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.222981:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.222983:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.222988:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.222990:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.222990:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.222991:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.222992:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.222994:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.223000:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23912us (24054us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.223006:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57635 00000100:00000040:4.0:1496184948.223008:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.223009:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.223010:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.223013:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.223018:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.223019:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.223020:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.223021:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.223022:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.223024:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.223026:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.223026:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.223027:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.223028:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00002000:00000001:4.0:1496184948.223029:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.223030:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.223032:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3f900. 00000020:00000010:4.0:1496184948.223033:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.223035:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.223036:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.223038:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.223040:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.223045:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.223046:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.233393:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.233395:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.233397:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5850 x1568861104105264 msgsize 520 00000100:00100000:7.0:1496184948.233402:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.233411:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.233413:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.233447:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.233449:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.233451:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.233452:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.233453:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.233454:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.233456:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.233458:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105264 00000020:00000001:6.0:1496184948.233460:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.233460:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.233461:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.233463:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.233464:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.233466:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.233468:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.233469:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.233470:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7ca00. 00000020:00000010:6.0:1496184948.233472:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26480. 00000020:00000010:6.0:1496184948.233474:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.233475:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.233476:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.233477:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.233479:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.233482:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c5850 x1568861104105264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.233488:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.233488:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233493:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.233495:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.233496:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.233497:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.233497:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.233498:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.233499:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.233501:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43380 00000100:00000040:6.0:1496184948.233503:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.233504:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676232784 : -131872033318832 : ffff8810264c5850) 00000100:00000040:6.0:1496184948.233506:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5850 x1568861104105264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.233512:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.233512:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.233514:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105264:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.233516:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105264 00000020:00000001:6.0:1496184948.233517:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.233519:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.233522:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.233525:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.233526:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.233527:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.233528:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233529:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.233532:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.233533:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.233535:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.233537:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.233538:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.233540:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.233543:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.233544:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.233545:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.233546:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.233547:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.233548:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.233549:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.233550:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.233552:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.233553:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.233554:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.233555:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.233556:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.233559:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.233560:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.233561:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.233563:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.233564:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.233566:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.233567:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.233568:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.233569:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.233571:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.233572:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.233574:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.233575:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.233576:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.233577:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.233579:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.233580:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.233581:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.233582:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.233583:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.233584:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.233585:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.233587:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.233588:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.233589:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233590:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.233593:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.233595:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.233596:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.233598:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.233599:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.233600:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233601:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.233603:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.233604:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.233605:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.233607:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.233608:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233609:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.233610:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.233611:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.233612:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.233613:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.233614:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.233615:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.233617:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.233618:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.233619:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.233621:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.233622:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.233623:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.233625:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.233626:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.233627:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.233627:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.233628:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.233629:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.233631:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.233632:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105264 00010000:00000001:6.0:1496184948.233633:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.233635:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c5850 x1568861104105264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.233640:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.233641:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.233642:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.233644:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.233646:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.233647:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.233649:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.233650:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.233651:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233652:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.233653:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.233655:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105264, offset 0 00000100:00000001:6.0:1496184948.233666:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.233667:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.233669:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.233670:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.233671:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.233671:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.233672:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.233674:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5850 x1568861104105264/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.233680:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105264:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (281us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.233684:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43380 00000100:00000040:6.0:1496184948.233686:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.233687:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.233687:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.233689:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26480. 00000020:00000010:6.0:1496184948.233691:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.233692:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7ca00. 00000020:00000040:6.0:1496184948.233693:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.233695:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.233697:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.233700:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.233701:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.233723:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.233725:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.233727:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77a000. 02000000:00000001:7.0:1496184948.233730:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.233730:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.234175:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.234177:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.234179:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026070850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.234184:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.234186:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.234188:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:4.0:1496184948.234219:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.234222:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.234224:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.234225:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.234226:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.234227:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.234229:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.234233:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.234235:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.234235:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.234237:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.234238:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.234239:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.234241:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.234243:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.234244:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.234246:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.234247:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3ff80. 00000020:00000010:4.0:1496184948.234249:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.234251:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.234252:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.234253:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.234254:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.234255:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.234256:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.234258:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.234259:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.234260:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.234261:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.234262:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.234263:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.234264:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.234265:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.234266:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.234267:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234268:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.234269:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234270:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234271:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.234272:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.234273:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.234275:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.234277:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.234284:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.234285:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.234287:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.234288:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.234289:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.234290:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.234292:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.234292:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.234293:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.234293:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00002000:00000001:4.0:1496184948.234295:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.234296:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.234298:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.234302:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.234304:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.234305:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.234306:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.234307:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.234308:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.234309:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.234311:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57636 00000100:00000040:4.0:1496184948.234313:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.234314:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671690320 : -131872037861296 : ffff881026070850) 00000100:00000040:4.0:1496184948.234316:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.234321:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.234322:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.234324:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.234326:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.234327:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.234329:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.234330:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234331:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.234332:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.234334:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.234335:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.234336:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.234336:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.234337:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234338:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.234340:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.234341:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.234342:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.234343:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.234344:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234346:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.234347:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.234348:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.234353:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.234355:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.234356:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.234358:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.234360:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.234361:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.234362:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.234363:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.234365:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.234366:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.234368:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.234369:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.234370:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.234371:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.234371:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.234373:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.234375:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.234376:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.234378:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.234400:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.234405:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234407:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.234410:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.234411:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.234412:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.234413:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.234414:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.234415:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.234417:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.234418:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.234423:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.234424:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.234450:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe740-0x592def0cfe740 00000100:00000001:4.0:1496184948.234452:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.236002:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.236005:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.247039:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.247045:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.247046:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184948.258066:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184948.258097:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184948.258099:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184948.258103:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c850 x1568861104105312 msgsize 520 00000100:00100000:1.0:1496184948.258110:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.258111:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.258114:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.258120:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.258123:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.258136:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 00010000:00020000:4.0:1496184948.258138:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 02000000:00000001:3.0:1496184948.258138:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.258140:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184948.258142:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:4.0:1496184948.258144:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:3.0:1496184948.258146:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.258147:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.258150:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.258151:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00100000:3.0:1496184948.258151:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105312 00002000:00000001:4.0:1496184948.258154:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00000020:00000001:3.0:1496184948.258154:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:4.0:1496184948.258155:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000040:3.0:1496184948.258155:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184948.258158:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184948.258160:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184948.258162:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184948.258164:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184948.258171:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.258172:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184948.258175:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000010:3.0:1496184948.258179:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132f780. 00000020:00000010:3.0:1496184948.258181:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184948.258184:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.258185:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184948.258187:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000001:00000001:4.0:1496184948.258188:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.258191:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000100:00000001:3.0:1496184948.258192:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00002000:00000001:4.0:1496184948.258193:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.258194:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.258194:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.258196:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.258197:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000200:3.0:1496184948.258197:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8c850 x1568861104105312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000001:4.0:1496184948.258198:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.258201:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.258202:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000100:00000001:3.0:1496184948.258202:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000020:00000040:4.0:1496184948.258203:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000001:3.0:1496184948.258204:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000010:4.0:1496184948.258205:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.258207:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.258208:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 00000100:00000001:3.0:1496184948.258208:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:4.0:1496184948.258209:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.258210:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 00004000:00000001:3.0:1496184948.258211:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 02000000:00000001:4.0:1496184948.258212:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00004000:00000001:3.0:1496184948.258213:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:02000400:4.0:1496184948.258214:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00004000:00000001:3.0:1496184948.258214:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.258214:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000020:00000001:4.0:1496184948.258215:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184948.258216:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184948.258217:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000080:4.0:1496184948.258218:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184948.258221:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6095 00010000:00000040:4.0:1496184948.258223:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:3.0:1496184948.258223:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00010000:00000001:4.0:1496184948.258224:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00000100:00000001:3.0:1496184948.258224:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736398416 : -131906973153200 : ffff880803b8c850) 00010000:00000001:4.0:1496184948.258225:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.258226:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.258227:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.258227:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c850 x1568861104105312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000040:4.0:1496184948.258229:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.258233:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.258234:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.258236:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23913us (24054us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184948.258236:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104105312:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184948.258239:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105312 00000100:00100000:4.0:1496184948.258241:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57636 00000020:00000001:3.0:1496184948.258241:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000100:00000040:4.0:1496184948.258243:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.258244:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.258245:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00010000:00000001:3.0:1496184948.258245:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00002000:00100000:4.0:1496184948.258248:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184948.258253:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00002000:00000001:4.0:1496184948.258254:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.258254:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.258256:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.258258:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:3.0:1496184948.258258:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 00000020:00000001:4.0:1496184948.258259:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 02000000:00000001:3.0:1496184948.258259:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 00010000:00000001:4.0:1496184948.258260:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.258262:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.258263:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.258263:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 02000000:00000010:3.0:1496184948.258263:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fd000. 00002000:00010000:4.0:1496184948.258264:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 02000000:00000001:3.0:1496184948.258264:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.258265:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.258267:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:3.0:1496184948.258267:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:4.0:1496184948.258268:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3ff80. 00000020:00000010:4.0:1496184948.258270:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.258272:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:3.0:1496184948.258272:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:4.0:1496184948.258273:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000001:4.0:1496184948.258275:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00010000:00000001:3.0:1496184948.258275:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00000100:00000200:4.0:1496184948.258277:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00010000:02000400:3.0:1496184948.258278:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.258280:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00004000:00000001:4.0:1496184948.258282:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00010000:00000001:3.0:1496184948.258282:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00004000:00000001:4.0:1496184948.258284:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00080000:3.0:1496184948.258285:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.258289:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.258292:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.258294:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.258296:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.258298:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.258299:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.258301:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.258303:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.258305:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.258307:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.258308:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.258310:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.258311:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.258316:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.258317:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.258319:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.258321:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.258323:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.258325:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.258334:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.258335:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.258336:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.258339:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.258341:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.258343:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.258344:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.258346:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.258348:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.258349:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.258351:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.258352:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.258354:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.258356:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.258357:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.258358:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.258360:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.258362:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.258364:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.258365:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.258369:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.258371:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.258373:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.258376:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.258377:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.258379:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.258380:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.258381:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.258383:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.258385:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.258387:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.258389:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.258391:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.258393:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.258394:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.258396:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.258398:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.258399:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.258401:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.258403:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.258404:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.258406:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.258408:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.258411:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.258413:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.258415:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.258416:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.258417:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.258418:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.258420:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.258422:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.258424:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.258426:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105312 00010000:00000001:3.0:1496184948.258428:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.258431:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8c850 x1568861104105312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.258438:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.258439:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.258441:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.258443:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.258445:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.258447:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.258448:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.258449:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.258451:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.258452:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.258454:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.258458:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105312, offset 0 00000100:00000001:3.0:1496184948.258484:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.258486:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.258488:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.258489:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.258490:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.258491:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.258492:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.258494:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c850 x1568861104105312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.258500:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105312:12345-10.37.248.124@o2ib1:8 Request procesed in 265us (396us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.258505:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6095 00000100:00000040:3.0:1496184948.258507:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.258508:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.258509:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.258512:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132f780. 00000020:00000010:3.0:1496184948.258514:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184948.258516:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000040:3.0:1496184948.258518:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.258520:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.258522:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.258528:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.258529:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.259321:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.259324:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.259326:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184948.259327:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184948.259328:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184948.259329:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184948.259332:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184948.259334:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184948.259336:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184948.259336:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184948.259338:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184948.259339:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184948.259341:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184948.259342:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184948.259345:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.259346:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184948.259347:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184948.259349:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3f380. 00000020:00000010:4.0:1496184948.259353:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184948.259355:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184948.259357:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184948.259358:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184948.259359:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184948.259360:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.259361:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.259363:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.259364:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184948.259365:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184948.259366:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.259367:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184948.259368:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.259369:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.259370:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.259371:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184948.259371:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259372:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184948.259374:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259375:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259384:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.259385:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184948.259386:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.259387:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.259390:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184948.259396:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.259397:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.259399:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.259400:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.259401:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.259402:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.259404:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.259404:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.259405:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.259406:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00002000:00000001:4.0:1496184948.259407:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.259409:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.259411:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.259416:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184948.259418:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.259419:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184948.259420:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184948.259421:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184948.259422:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184948.259423:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184948.259425:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57637 00000100:00000040:4.0:1496184948.259427:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184948.259428:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671688272 : -131872037863344 : ffff881026070050) 00000100:00000040:4.0:1496184948.259431:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184948.259436:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184948.259436:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184948.259438:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184948.259440:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184948.259442:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184948.259443:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184948.259444:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259445:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184948.259446:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184948.259447:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184948.259449:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184948.259450:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184948.259450:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184948.259451:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259452:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184948.259454:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184948.259455:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184948.259456:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.259458:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.259459:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259461:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184948.259462:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184948.259464:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184948.259468:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184948.259470:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184948.259471:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.259474:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.259475:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184948.259476:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184948.259481:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184948.259482:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184948.259484:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184948.259485:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184948.259487:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184948.259488:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184948.259489:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184948.259490:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184948.259491:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184948.259493:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184948.259493:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184948.259494:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184948.259497:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184948.259518:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184948.259523:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259525:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184948.259527:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.259529:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184948.259530:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.259531:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.259532:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184948.259533:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184948.259535:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184948.259536:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184948.259541:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184948.259542:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184948.259568:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe770-0x592def0cfe770 00000100:00000001:4.0:1496184948.259570:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.269962:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.269964:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:6.0:1496184948.274943:0:13696:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184948.274945:0:13696:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.274949:0:13696:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184948.274950:0:13696:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.274951:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184948.274953:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.274954:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184948.274955:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184948.277950:0:13694:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:7.0:1496184948.277952:0:13694:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:7.0:1496184948.277956:0:13694:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:7.0:1496184948.277957:0:13694:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184948.277958:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184948.277960:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184948.277961:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184948.277962:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.280937:0:13722:0:(ldlm_pool.c:339:ldlm_srv_pool_recalc()) Process entered 00010000:00000001:6.0:1496184948.280939:0:13722:0:(ldlm_pool.c:371:ldlm_srv_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.280945:0:13722:0:(ldlm_pool.c:478:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:6.0:1496184948.280945:0:13722:0:(ldlm_pool.c:482:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.280952:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184948.280954:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.280956:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184948.280957:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.280969:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.280973:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.280974:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000020:00000001:1.0:1496184948.280980:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184948.280982:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184948.280984:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184948.280984:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.290944:0:13693:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184948.290946:0:13693:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.290950:0:13693:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184948.290951:0:13693:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.290952:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184948.290954:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.290955:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184948.290956:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.291957:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184948.292033:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184948.292039:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.292042:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184948.292045:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184948.292046:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184948.292048:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184948.292050:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184948.292083:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184948.292085:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184948.292087:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184948.292088:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184948.292089:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184948.292090:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.292091:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184948.292093:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184948.292094:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184948.292094:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184948.292096:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184948.292097:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184948.292099:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184948.292100:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184948.292101:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184948.292102:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184948.292103:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184948.292105:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184948.292107:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184948.292111:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184948.292117:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184948.292118:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184948.292119:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184948.292120:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184948.292121:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184948.292123:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184948.292129:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32692us (32845us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184948.292134:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57637 00000100:00000040:4.0:1496184948.292136:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184948.292137:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184948.292138:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184948.292141:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184948.292146:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184948.292147:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184948.292149:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184948.292150:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184948.292151:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.292152:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184948.292154:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184948.292154:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184948.292155:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184948.292156:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00002000:00000001:4.0:1496184948.292157:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184948.292158:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184948.292160:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3f380. 00000020:00000010:4.0:1496184948.292162:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184948.292164:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184948.292165:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184948.292167:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184948.292169:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184948.292174:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184948.292175:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:1.0:1496184948.303020:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184948.303022:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184948.303024:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8e450 x1568861104105344 msgsize 520 00000100:00100000:1.0:1496184948.303030:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.303032:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.303035:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.303069:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.303072:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.303075:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184948.303077:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.303078:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184948.303080:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184948.303084:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184948.303089:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105344 00000020:00000001:3.0:1496184948.303092:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184948.303093:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184948.303095:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184948.303097:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:3.0:1496184948.303099:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184948.303100:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184948.303104:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.303105:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184948.303107:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000010:3.0:1496184948.303110:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132f100. 00000020:00000010:3.0:1496184948.303112:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184948.303114:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.303115:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184948.303116:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184948.303120:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184948.303124:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8e450 x1568861104105344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184948.303129:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184948.303130:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303136:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184948.303139:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.303140:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184948.303141:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.303142:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.303143:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184948.303144:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184948.303147:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6096 00000100:00000040:3.0:1496184948.303149:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.303150:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736405584 : -131906973146032 : ffff880803b8e450) 00000100:00000040:3.0:1496184948.303153:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8e450 x1568861104105344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184948.303158:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.303159:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184948.303162:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105344:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184948.303164:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105344 00000020:00000001:3.0:1496184948.303166:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184948.303169:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184948.303174:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184948.303178:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184948.303179:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184948.303181:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fdc00. 02000000:00000001:3.0:1496184948.303183:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303185:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.303188:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184948.303190:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184948.303192:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.303193:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184948.303195:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184948.303197:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.303200:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.303202:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.303203:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.303205:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.303206:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.303207:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.303209:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.303210:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.303212:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.303213:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.303214:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.303216:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.303217:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.303221:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.303222:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.303224:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.303227:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.303228:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.303230:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.303232:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.303233:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.303234:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.303236:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.303237:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.303239:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.303240:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.303241:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.303243:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.303245:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.303246:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.303247:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.303249:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.303250:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.303251:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.303252:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.303254:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.303256:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.303257:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303258:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.303261:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.303263:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.303265:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.303267:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.303269:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.303270:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303271:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.303272:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.303274:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.303276:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.303278:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.303279:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303281:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.303282:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.303284:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.303285:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.303287:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.303288:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.303289:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.303291:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.303292:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.303293:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.303295:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.303297:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.303298:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.303300:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.303301:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.303302:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.303303:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.303305:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.303306:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.303307:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.303309:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105344 00010000:00000001:3.0:1496184948.303311:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.303313:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8e450 x1568861104105344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.303319:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.303320:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.303321:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.303323:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.303325:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.303327:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.303328:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.303330:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.303331:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303332:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.303334:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.303337:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105344, offset 0 00000100:00000001:3.0:1496184948.303351:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.303352:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.303353:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.303354:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.303355:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.303356:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.303357:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.303359:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8e450 x1568861104105344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.303365:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105344:12345-10.37.248.124@o2ib1:8 Request procesed in 205us (339us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.303370:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6096 00000100:00000040:3.0:1496184948.303372:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.303373:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.303374:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.303376:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132f100. 00000020:00000010:3.0:1496184948.303378:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184948.303380:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000040:3.0:1496184948.303381:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.303383:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.303385:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.303389:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.303390:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.303906:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.303908:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.303910:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.303911:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.303912:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.303913:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.303916:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.303918:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.303920:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.303921:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.303922:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.303923:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.303925:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.303927:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.303929:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.303930:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.303932:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.303934:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26d80. 00000020:00000010:6.0:1496184948.303936:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.303939:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.303940:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.303941:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.303942:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.303943:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.303944:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.303946:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.303947:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.303948:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.303949:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.303950:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.303951:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.303952:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.303953:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.303954:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.303954:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.303956:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.303957:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.303958:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.303959:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.303961:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.303962:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.303963:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.303966:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.303972:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.303973:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.303975:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.303975:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.303976:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.303978:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.303979:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.303980:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.303981:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.303981:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00002000:00000001:6.0:1496184948.303982:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.303984:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.303986:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.303990:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.303993:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.303994:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.303995:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.303995:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.303996:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.303997:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.304000:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57638 00000100:00000040:6.0:1496184948.304001:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.304003:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671270480 : -131872038281136 : ffff88102600a050) 00000100:00000040:6.0:1496184948.304005:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.304010:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.304011:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.304013:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.304015:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.304016:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.304018:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.304019:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.304020:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.304021:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.304022:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.304023:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.304024:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.304025:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.304026:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.304027:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.304029:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.304030:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.304032:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.304033:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.304034:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.304036:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.304036:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.304038:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.304043:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.304045:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.304046:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.304048:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.304050:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.304051:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.304052:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.304054:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.304056:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.304057:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.304059:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.304061:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.304061:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.304062:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.304063:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.304065:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.304065:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.304066:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.304069:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.304092:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.304098:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.304100:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.304103:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.304104:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.304105:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.304106:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.304107:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.304108:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c800. 00000100:00000010:6.0:1496184948.304110:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.304111:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.304116:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.304117:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.304143:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe790-0x592def0cfe790 00000100:00000001:6.0:1496184948.304146:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.305888:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.305914:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184948.305920:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.305925:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.305928:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c800 00000100:00000001:3.0:1496184948.305933:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.305973:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.305979:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.305981:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.305985:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.305986:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.305988:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.305990:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.306023:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.306025:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.306027:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.306028:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.306029:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.306030:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.306031:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.306033:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.306033:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.306034:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.306036:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.306037:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.306039:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c800. 00000100:00000001:6.0:1496184948.306040:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.306041:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.306042:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.306043:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.306045:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.306047:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.306049:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.306054:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.306056:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.306057:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.306057:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.306059:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.306061:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.306067:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2055us (2210us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.306072:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57638 00000100:00000040:6.0:1496184948.306073:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.306075:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.306076:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.306078:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.306084:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.306084:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.306086:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.306087:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.306088:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.306090:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.306091:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.306092:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.306092:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.306093:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00002000:00000001:6.0:1496184948.306094:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.306096:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.306097:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26d80. 00000020:00000010:6.0:1496184948.306099:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.306101:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.306102:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.306103:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.306106:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.306111:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.306112:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.335038:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.335041:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.335042:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.335043:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.335044:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.335045:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.335048:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.335050:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105376 00000020:00000001:6.0:1496184948.335051:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.335052:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.335053:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.335054:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.335056:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.335058:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.335060:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.335061:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.335062:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.335064:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880807b26e00. 00000020:00000010:6.0:1496184948.335066:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.335067:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.335068:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.335069:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.335071:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.335074:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6450 x1568861104105376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.335080:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.335080:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335085:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.335087:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.335088:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.335089:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.335089:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.335090:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.335091:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.335093:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43381 00000100:00000040:6.0:1496184948.335095:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.335096:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676235856 : -131872033315760 : ffff8810264c6450) 00000100:00000040:6.0:1496184948.335098:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6450 x1568861104105376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.335103:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.335104:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.335106:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105376:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.335108:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105376 00000020:00000001:6.0:1496184948.335109:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.335111:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.335114:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.335117:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.335118:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.335120:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.335121:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335122:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.335125:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.335126:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.335128:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.335130:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.335131:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.335133:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.335136:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.335138:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.335139:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.335140:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.335141:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.335142:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.335143:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.335144:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.335146:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.335147:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.335148:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.335149:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.335151:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.335154:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.335155:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.335156:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.335158:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.335159:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.335161:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.335162:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.335163:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.335164:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.335166:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.335168:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.335169:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.335171:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.335172:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.335173:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.335175:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.335176:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.335177:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.335178:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.335180:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.335181:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.335182:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.335183:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.335185:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.335186:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335187:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.335190:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.335192:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.335194:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.335196:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.335197:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.335198:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335199:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.335200:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.335202:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.335203:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.335204:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.335206:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335207:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.335208:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.335209:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.335210:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.335212:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.335213:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.335214:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.335216:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.335217:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.335218:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.335219:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335221:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.335222:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.335223:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.335224:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.335225:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.335226:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.335227:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335228:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.335229:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.335231:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105376 00010000:00000001:6.0:1496184948.335232:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.335234:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6450 x1568861104105376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.335239:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.335240:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.335241:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.335244:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.335245:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.335247:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.335248:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.335249:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.335250:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335251:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.335253:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.335256:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105376, offset 0 00000100:00000001:6.0:1496184948.335267:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335268:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.335269:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.335270:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.335271:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.335272:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335273:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.335275:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6450 x1568861104105376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.335280:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105376:12345-10.37.248.124@o2ib1:8 Request procesed in 176us (284us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.335285:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43381 00000100:00000040:6.0:1496184948.335286:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.335288:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.335288:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.335290:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880807b26e00. 00000020:00000010:6.0:1496184948.335292:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.335293:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.335294:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.335296:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.335298:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.335301:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.335302:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.335857:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.335860:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.335862:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.335863:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.335864:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.335865:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.335868:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.335870:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.335871:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.335872:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.335873:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.335874:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.335876:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.335878:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.335880:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.335881:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.335882:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.335886:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684800. 00000020:00000010:6.0:1496184948.335888:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.335901:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.335903:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.335903:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.335904:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.335906:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.335907:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.335908:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.335909:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.335910:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.335911:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.335913:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.335913:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.335914:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.335915:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.335916:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.335917:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335918:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.335919:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335920:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335921:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.335922:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.335923:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.335924:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.335927:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.335934:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.335934:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.335936:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.335937:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.335938:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.335940:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.335941:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.335942:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.335942:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.335943:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00002000:00000001:6.0:1496184948.335944:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335945:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335947:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335952:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.335954:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.335955:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.335956:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.335956:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.335957:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.335958:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.335961:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57639 00000100:00000040:6.0:1496184948.335962:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.335963:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671689296 : -131872037862320 : ffff881026070450) 00000100:00000040:6.0:1496184948.335966:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.335971:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.335971:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.335973:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.335975:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.335976:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.335979:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.335979:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335981:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.335981:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.335983:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.335984:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.335985:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.335986:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.335986:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335987:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.335989:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.335990:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.335991:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e778000. 02000000:00000001:6.0:1496184948.335992:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.335994:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.335995:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.335996:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.335998:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.336002:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.336005:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.336005:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.336008:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.336009:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.336010:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.336011:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.336013:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.336015:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.336015:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.336017:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.336019:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.336019:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.336020:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.336021:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.336023:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.336023:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.336024:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.336027:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.336048:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.336054:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.336056:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.336059:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.336060:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.336061:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.336062:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.336063:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.336064:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7de00. 00000100:00000010:6.0:1496184948.336066:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.336067:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.336072:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.336073:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.336099:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe7b0-0x592def0cfe7b0 00000100:00000001:6.0:1496184948.336102:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.337732:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.337734:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.337753:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.337757:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.337758:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.337763:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.337802:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.337808:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.337811:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.337814:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.337815:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.337817:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.337819:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.337852:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.337854:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.337856:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.337857:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.337858:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.337859:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.337860:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.337862:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.337862:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.337863:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.337864:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.337866:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.337867:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7de00. 00000100:00000001:6.0:1496184948.337869:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.337870:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.337871:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e778000. 02000000:00000001:6.0:1496184948.337872:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.337874:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.337875:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.337878:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.337883:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.337884:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.337885:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.337886:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.337887:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.337889:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.337903:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1930us (2084us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.337908:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57639 00000100:00000040:6.0:1496184948.337909:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.337911:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.337912:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.337914:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.337920:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.337920:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.337922:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.337923:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.337924:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.337926:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.337927:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.337928:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.337929:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.337929:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00002000:00000001:6.0:1496184948.337931:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.337932:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.337933:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684800. 00000020:00000010:6.0:1496184948.337935:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.337936:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.337938:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.337939:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.337942:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.337947:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.337948:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:1.0:1496184948.341933:0:13689:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184948.341936:0:13689:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184948.341938:0:13690:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1496184948.341940:0:13690:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184948.341943:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184948.341944:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.341945:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184948.341946:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184948.341946:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.341947:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184948.341949:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.341949:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184948.341950:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184948.341951:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.341951:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184948.341952:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.341953:0:13691:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1496184948.341955:0:13691:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184948.341960:0:13691:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184948.341961:0:13691:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.341962:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184948.341964:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.341965:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184948.341966:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184948.350929:0:13692:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184948.350930:0:13692:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184948.350933:0:13692:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184948.350934:0:13692:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184948.350935:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184948.350936:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184948.350937:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184948.350938:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184948.366750:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.366752:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.366755:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8450 x1568861104105408 msgsize 520 00000100:00100000:7.0:1496184948.366760:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.366761:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.366763:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.366794:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.366797:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.366798:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.366799:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.366800:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.366801:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.366804:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.366806:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105408 00000020:00000001:6.0:1496184948.366807:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.366808:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.366809:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.366811:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.366812:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.366814:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.366816:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.366816:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.366818:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.366820:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684e00. 00000020:00000010:6.0:1496184948.366822:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.366823:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.366824:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.366825:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.366827:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.366831:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8450 x1568861104105408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.366836:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.366836:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.366841:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.366843:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.366844:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.366845:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.366845:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.366846:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.366847:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.366849:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43382 00000100:00000040:6.0:1496184948.366851:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.366852:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677292624 : -131872032258992 : ffff8810265c8450) 00000100:00000040:6.0:1496184948.366855:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8450 x1568861104105408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.366860:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.366860:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.366862:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105408:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.366864:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105408 00000020:00000001:6.0:1496184948.366865:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.366867:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.366870:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.366873:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.366874:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.366875:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778000. 02000000:00000001:6.0:1496184948.366876:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.366878:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.366880:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.366882:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.366883:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.366885:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.366886:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.366889:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.366901:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.366903:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.366903:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.366905:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.366906:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.366906:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.366907:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.366909:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.366910:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.366911:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.366912:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.366913:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.366914:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.366918:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.366918:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.366920:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.366922:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.366922:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.366924:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.366926:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.366927:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.366928:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.366930:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.366931:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.366933:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.366934:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.366935:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.366936:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.366938:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.366939:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.366940:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.366941:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.366942:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.366943:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.366944:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.366946:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.366947:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.366948:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.366950:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.366952:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.366954:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.366956:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.366957:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.366958:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.366959:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.366961:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.366962:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.366963:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.366965:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.366966:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.366967:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.366968:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.366969:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.366970:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.366971:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.366972:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.366973:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.366974:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.366976:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.366977:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.366978:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.366979:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.366981:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.366982:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.366984:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.366984:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.366985:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.366986:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.366987:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.366988:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.366989:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.366990:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105408 00010000:00000001:6.0:1496184948.366992:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.366994:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8450 x1568861104105408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.366999:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.367000:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.367001:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.367003:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.367005:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.367006:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.367007:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.367009:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.367010:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367011:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.367012:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.367014:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105408, offset 0 00000100:00000001:6.0:1496184948.367025:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367026:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.367028:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.367029:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.367030:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.367030:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367031:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.367033:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8450 x1568861104105408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.367043:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105408:12345-10.37.248.124@o2ib1:8 Request procesed in 182us (287us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.367047:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43382 00000100:00000040:6.0:1496184948.367049:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.367050:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.367051:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.367052:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684e00. 00000020:00000010:6.0:1496184948.367054:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.367056:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.367057:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.367058:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.367060:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.367064:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.367064:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.367088:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.367090:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.367092:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e778000. 02000000:00000001:7.0:1496184948.367095:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.367095:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.367502:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.367504:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.367506:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.367511:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.367512:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.367514:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.367545:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.367548:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.367550:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.367551:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.367552:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.367553:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.367555:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.367557:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.367559:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.367559:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.367561:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.367562:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.367563:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.367565:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.367567:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.367568:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.367569:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.367571:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684000. 00000020:00000010:6.0:1496184948.367573:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.367575:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.367577:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.367577:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.367578:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.367579:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.367580:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.367582:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.367583:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.367584:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.367585:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.367586:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.367587:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.367588:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.367589:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.367590:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.367590:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367591:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.367593:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367594:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367595:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.367596:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.367597:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.367598:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.367601:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.367607:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.367608:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.367610:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.367610:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.367611:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.367613:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.367614:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.367615:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.367616:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.367616:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00002000:00000001:6.0:1496184948.367618:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367619:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367621:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367625:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.367627:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.367628:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.367629:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.367630:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.367631:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.367632:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.367634:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57640 00000100:00000040:6.0:1496184948.367636:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.367637:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671269456 : -131872038282160 : ffff881026009c50) 00000100:00000040:6.0:1496184948.367639:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.367645:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.367645:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.367647:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.367649:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.367650:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.367652:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.367653:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367654:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.367654:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.367656:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.367657:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.367658:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.367659:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.367659:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367660:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.367662:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.367663:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.367665:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e778000. 02000000:00000001:6.0:1496184948.367666:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367667:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367669:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.367669:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.367671:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.367676:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.367678:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.367679:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.367681:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.367683:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.367684:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.367685:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.367686:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.367688:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.367689:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.367691:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.367692:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.367693:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.367693:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.367694:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.367696:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.367697:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.367698:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.367700:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.367723:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.367729:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367730:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.367733:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.367734:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.367735:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.367736:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.367737:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.367738:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e600. 00000100:00000010:6.0:1496184948.367740:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.367741:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.367746:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.367747:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.367773:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe7d0-0x592def0cfe7d0 00000100:00000001:6.0:1496184948.367775:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.369279:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.369281:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184948.369287:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.369292:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.369293:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184948.369297:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.369336:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.369351:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.369353:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.369357:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.369357:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.369360:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.369361:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.369394:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.369397:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.369399:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.369400:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.369400:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.369402:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.369403:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.369404:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.369405:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.369405:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.369407:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.369408:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.369410:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e600. 00000100:00000001:6.0:1496184948.369411:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.369412:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.369413:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e778000. 02000000:00000001:6.0:1496184948.369414:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.369416:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.369418:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.369420:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.369426:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.369427:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.369428:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.369429:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.369430:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.369432:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.369438:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1792us (1930us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.369443:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57640 00000100:00000040:6.0:1496184948.369444:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.369446:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.369447:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.369449:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.369454:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.369455:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.369457:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.369458:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.369459:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.369460:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.369462:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.369463:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.369463:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.369464:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00002000:00000001:6.0:1496184948.369465:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.369466:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.369468:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684000. 00000020:00000010:6.0:1496184948.369469:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.369471:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.369473:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.369474:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.369476:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.369481:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.369482:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:1.0:1496184948.391928:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184948.391932:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184948.391936:0:14306:0:(service.c:1448:ptlrpc_at_check_timed()) Process entered 00000100:00000001:1.0:1496184948.391938:0:14306:0:(service.c:1460:ptlrpc_at_check_timed()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:1.0:1496184948.391941:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184948.391943:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.397587:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.397589:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.397591:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.397592:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.397593:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.397594:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.397596:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.397598:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105440 00000020:00000001:6.0:1496184948.397600:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.397601:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.397602:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.397603:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.397604:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.397606:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.397608:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.397609:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.397611:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.397612:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684080. 00000020:00000010:6.0:1496184948.397614:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.397616:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.397616:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.397617:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.397619:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.397622:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8850 x1568861104105440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.397627:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.397628:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397633:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.397635:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.397636:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.397636:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.397637:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.397638:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.397639:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.397641:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43383 00000100:00000040:6.0:1496184948.397643:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.397644:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677293648 : -131872032257968 : ffff8810265c8850) 00000100:00000040:6.0:1496184948.397646:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8850 x1568861104105440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.397652:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.397652:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.397654:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105440:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.397656:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105440 00000020:00000001:6.0:1496184948.397657:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.397659:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.397662:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.397665:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.397666:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.397667:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778000. 02000000:00000001:6.0:1496184948.397668:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397670:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.397672:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.397674:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.397675:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.397677:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.397678:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.397680:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.397683:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.397684:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.397685:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.397686:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.397687:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.397688:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.397689:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.397690:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.397691:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.397692:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.397694:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.397695:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.397696:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.397699:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.397700:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.397701:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.397703:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.397704:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.397706:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.397707:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.397708:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.397709:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.397711:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.397712:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.397714:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.397715:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.397715:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.397717:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.397719:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.397720:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.397720:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.397722:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.397723:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.397724:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.397725:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.397726:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.397728:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.397729:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397730:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.397733:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.397735:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.397736:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.397738:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.397739:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.397740:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397741:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.397743:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.397744:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.397745:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.397746:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.397748:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397749:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.397750:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.397751:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.397752:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.397753:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.397754:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.397755:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.397757:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.397758:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.397759:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.397760:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.397762:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.397763:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.397764:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.397765:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.397766:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.397767:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.397768:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.397769:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.397770:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.397771:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105440 00010000:00000001:6.0:1496184948.397773:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.397775:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8850 x1568861104105440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.397780:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.397781:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.397782:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.397784:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.397786:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.397787:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.397788:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.397790:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.397790:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397791:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.397793:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.397795:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105440, offset 0 00000100:00000001:6.0:1496184948.397806:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.397807:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.397808:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.397809:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.397810:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.397811:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.397812:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.397814:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8850 x1568861104105440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.397819:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105440:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.397824:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43383 00000100:00000040:6.0:1496184948.397825:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.397826:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.397827:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.397829:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684080. 00000020:00000010:6.0:1496184948.397831:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.397832:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.397833:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.397835:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.397837:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.397840:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.397841:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.398372:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.398374:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.398376:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.398377:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.398378:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.398379:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.398381:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.398383:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.398385:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.398385:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.398387:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.398388:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.398389:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.398391:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.398393:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.398394:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.398396:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000010:6.0:1496184948.398397:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684900. 00000020:00000010:6.0:1496184948.398399:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.398402:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.398403:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.398404:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.398405:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.398406:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.398407:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.398408:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.398409:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.398411:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.398412:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.398413:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.398414:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.398415:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.398416:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.398417:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.398417:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398418:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.398419:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398420:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398421:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.398423:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.398424:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.398425:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.398428:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.398434:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.398435:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.398437:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.398437:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.398438:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.398440:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.398441:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.398442:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.398442:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.398443:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00002000:00000001:6.0:1496184948.398444:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.398445:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.398448:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.398452:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.398454:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.398455:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.398456:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.398456:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.398457:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.398458:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.398461:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57641 00000100:00000040:6.0:1496184948.398462:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.398463:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671274576 : -131872038277040 : ffff88102600b050) 00000100:00000040:6.0:1496184948.398466:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.398471:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.398472:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.398473:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.398475:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.398476:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.398478:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.398479:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398480:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.398481:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.398482:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.398483:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.398484:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.398485:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.398485:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398487:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.398488:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.398489:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.398491:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.398492:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.398493:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398494:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.398495:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.398497:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.398501:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.398504:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.398504:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.398507:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.398508:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.398509:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.398510:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.398512:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.398514:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.398514:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.398516:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.398518:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.398518:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.398519:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.398520:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.398522:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.398522:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.398523:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.398526:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.398547:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.398552:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398554:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.398556:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.398558:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.398559:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.398560:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.398561:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.398562:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7ec00. 00000100:00000010:6.0:1496184948.398563:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.398565:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.398570:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.398571:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.398598:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe7f0-0x592def0cfe7f0 00000100:00000001:6.0:1496184948.398600:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.400117:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.400131:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184948.411245:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.411250:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.411251:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7ec00 00000100:00000001:3.0:1496184948.422251:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.422317:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.422337:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.422339:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.422343:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.422344:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.422346:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.422347:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.422381:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.422383:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.422385:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.422386:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.422386:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.422388:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.422389:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.422390:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.422391:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.422392:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.422393:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.422394:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.422396:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7ec00. 00000100:00000001:6.0:1496184948.422397:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.422398:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.422399:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.422401:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.422402:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.422404:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.422407:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.422412:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.422413:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.422414:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.422414:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.422416:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.422418:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.422428:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23955us (24094us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.422433:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57641 00000100:00000040:6.0:1496184948.422434:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.422436:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.422437:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.422439:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.422445:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.422445:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.422447:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.422448:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.422449:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.422451:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.422452:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.422453:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.422454:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.422454:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00002000:00000001:6.0:1496184948.422455:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.422457:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.422458:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684900. 00000020:00000010:6.0:1496184948.422460:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.422461:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e200. 00000020:00000040:6.0:1496184948.422463:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.422464:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.422466:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.422472:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.422473:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.433090:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.433094:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.433097:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.433099:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.433100:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.433102:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.433105:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.433108:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105472 00000020:00000001:6.0:1496184948.433110:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.433112:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.433113:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.433115:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.433117:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.433120:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.433123:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.433124:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.433126:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7ec00. 00000020:00000010:6.0:1496184948.433129:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684780. 00000020:00000010:6.0:1496184948.433131:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.433145:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.433145:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.433146:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.433148:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.433151:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c5050 x1568861104105472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.433157:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.433157:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433162:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.433164:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.433165:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.433166:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.433166:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.433167:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.433168:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.433170:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43384 00000100:00000040:6.0:1496184948.433172:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.433173:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676230736 : -131872033320880 : ffff8810264c5050) 00000100:00000040:6.0:1496184948.433176:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5050 x1568861104105472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.433181:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.433181:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.433183:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105472:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.433185:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105472 00000020:00000001:6.0:1496184948.433186:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.433189:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.433191:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.433194:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.433195:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.433196:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779c00. 02000000:00000001:6.0:1496184948.433197:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433199:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.433201:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.433203:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.433204:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.433206:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.433207:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.433210:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.433212:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.433213:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.433214:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.433215:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.433216:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.433217:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.433218:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.433219:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.433221:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.433222:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.433223:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.433224:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.433225:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.433228:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.433229:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.433230:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.433232:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.433233:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.433235:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.433236:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.433237:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.433238:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.433240:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.433241:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.433243:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.433244:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.433245:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.433246:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.433248:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.433249:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.433249:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.433251:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.433252:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.433253:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.433254:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.433256:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.433257:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.433258:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433259:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.433262:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.433264:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.433265:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.433267:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.433268:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.433269:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433270:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.433272:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.433273:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.433274:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.433276:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.433277:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433278:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.433279:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.433280:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.433281:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.433282:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.433283:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.433284:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.433286:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.433287:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.433288:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.433289:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433291:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.433292:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.433293:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.433294:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.433295:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.433296:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.433297:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433298:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.433299:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.433300:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105472 00010000:00000001:6.0:1496184948.433302:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.433304:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c5050 x1568861104105472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.433309:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.433310:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.433311:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.433313:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.433315:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.433316:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.433317:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.433318:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.433319:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433320:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.433322:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.433324:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105472, offset 0 00000100:00000001:6.0:1496184948.433335:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433336:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.433337:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.433338:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.433339:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.433340:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433341:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.433343:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5050 x1568861104105472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.433348:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105472:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (311us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.433352:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43384 00000100:00000040:6.0:1496184948.433354:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.433355:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.433356:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.433357:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684780. 00000020:00000010:6.0:1496184948.433359:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.433360:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7ec00. 00000020:00000040:6.0:1496184948.433362:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.433363:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.433365:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.433368:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.433369:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.433439:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.433442:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.433445:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779c00. 02000000:00000001:7.0:1496184948.433449:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.433450:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.433831:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.433833:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.433835:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.433840:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.433841:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.433844:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.433877:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.433880:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.433881:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.433882:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.433883:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.433884:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.433887:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.433889:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.433904:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.433905:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.433906:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.433907:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.433909:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.433911:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.433913:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.433913:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.433915:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7ec00. 00000020:00000010:6.0:1496184948.433917:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684700. 00000020:00000010:6.0:1496184948.433918:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.433920:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.433922:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.433923:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.433924:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.433925:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.433926:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.433927:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.433928:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.433930:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.433930:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.433932:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.433933:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.433934:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.433934:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.433935:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.433936:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433937:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.433938:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433939:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433940:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.433942:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.433943:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.433944:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.433947:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.433953:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.433954:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.433956:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.433956:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.433957:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.433959:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.433960:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.433961:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.433961:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.433962:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00002000:00000001:6.0:1496184948.433963:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433965:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433967:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.433971:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.433973:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.433974:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.433975:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.433975:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.433976:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.433977:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.433980:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57642 00000100:00000040:6.0:1496184948.433981:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.433982:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671271504 : -131872038280112 : ffff88102600a450) 00000100:00000040:6.0:1496184948.433985:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.433990:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.433991:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.433993:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.433995:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.433995:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.433997:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.433998:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.433999:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.434000:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.434001:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.434002:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.434003:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.434004:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.434005:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.434006:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.434008:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.434008:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.434010:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.434011:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.434012:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.434014:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.434014:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.434016:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.434020:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.434023:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.434024:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.434026:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.434028:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.434029:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.434029:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.434031:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.434033:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.434033:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.434035:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.434037:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.434038:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.434038:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.434039:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.434041:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.434041:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.434042:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.434045:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.434068:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.434074:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.434075:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.434078:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.434079:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.434080:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.434081:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.434082:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.434084:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e000. 00000100:00000010:6.0:1496184948.434085:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.434086:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.434091:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.434092:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.434118:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe810-0x592def0cfe810 00000100:00000001:6.0:1496184948.434120:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.435645:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.435648:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184948.446712:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.446717:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.446718:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e000 00000100:00000001:3.0:1496184948.457754:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184948.457793:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184948.457796:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184948.457799:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8d450 x1568861104105504 msgsize 520 00000100:00100000:1.0:1496184948.457806:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.457807:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.457810:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.457817:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.457819:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00010000:00020000:6.0:1496184948.457821:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000001:3.0:1496184948.457824:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184948.457835:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.457837:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00010000:00000001:6.0:1496184948.457840:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184948.457841:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.457842:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.457846:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 02000000:00000001:3.0:1496184948.457846:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.457847:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.457849:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00000100:00100000:3.0:1496184948.457849:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105504 00002000:00000001:6.0:1496184948.457851:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1496184948.457852:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184948.457854:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184948.457856:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184948.457859:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184948.457861:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184948.457863:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184948.457867:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.457868:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184948.457871:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000010:3.0:1496184948.457875:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132f500. 00000020:00000010:3.0:1496184948.457878:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184948.457881:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.457882:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000001:00000001:6.0:1496184948.457884:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.457884:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000020:00000002:6.0:1496184948.457886:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.457888:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00000100:00000001:3.0:1496184948.457889:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00002000:00000001:6.0:1496184948.457899:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.457900:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000200:3.0:1496184948.457900:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8d450 x1568861104105504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.457901:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.457903:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.457904:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.457905:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.457905:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000100:00000001:3.0:1496184948.457905:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000020:00000040:6.0:1496184948.457907:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000001:3.0:1496184948.457907:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000010:6.0:1496184948.457908:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.457910:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e000. 00000100:00000001:3.0:1496184948.457910:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.457911:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.457912:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.457914:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a000. 00004000:00000001:3.0:1496184948.457914:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 02000000:00000001:6.0:1496184948.457915:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00004000:00000001:3.0:1496184948.457915:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184948.457916:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00000020:02000400:6.0:1496184948.457917:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00004000:00000001:3.0:1496184948.457917:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000020:00000001:6.0:1496184948.457918:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184948.457919:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184948.457920:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000080:6.0:1496184948.457921:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:3.0:1496184948.457924:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6097 00010000:00000040:6.0:1496184948.457926:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:3.0:1496184948.457926:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00010000:00000001:6.0:1496184948.457928:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.457928:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000100:00000001:3.0:1496184948.457928:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736401488 : -131906973150128 : ffff880803b8d450) 00000020:00000001:6.0:1496184948.457929:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.457930:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.457931:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8d450 x1568861104105504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000040:6.0:1496184948.457933:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.457939:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23947us (24102us total) trans 0 rc -110/-999 00000100:00000001:3.0:1496184948.457942:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.457943:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.457944:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57642 00000100:00100000:3.0:1496184948.457945:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104105504:12345-10.37.248.124@o2ib1:8 00000100:00000040:6.0:1496184948.457946:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.457948:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000200:3.0:1496184948.457948:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105504 00002000:00000001:6.0:1496184948.457949:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000020:00000001:3.0:1496184948.457950:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00002000:00100000:6.0:1496184948.457952:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00010000:00000001:3.0:1496184948.457955:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00002000:00000001:6.0:1496184948.457957:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.457958:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.457959:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.457961:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.457962:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000020:00000040:3.0:1496184948.457962:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00010000:00000001:6.0:1496184948.457963:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.457965:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.457966:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.457966:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.457967:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00000100:00000001:3.0:1496184948.457967:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 00002000:00000001:6.0:1496184948.457968:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184948.457969:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 00000100:00000001:6.0:1496184948.457970:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.457971:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684700. 02000000:00000010:3.0:1496184948.457971:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23f8c00. 00000020:00000010:6.0:1496184948.457973:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 02000000:00000001:3.0:1496184948.457973:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:6.0:1496184948.457975:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7ec00. 00000100:00000001:3.0:1496184948.457975:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.457976:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:6.0:1496184948.457978:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.457980:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000020:00000040:3.0:1496184948.457981:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184948.457983:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00004000:00000001:6.0:1496184948.457985:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.457986:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:02000400:3.0:1496184948.457986:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.457988:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184948.457990:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184948.457993:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.457997:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.458000:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.458002:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.458004:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.458006:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.458007:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.458009:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.458011:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.458013:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.458014:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.458016:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.458017:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.458019:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.458025:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.458026:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.458027:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.458030:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.458031:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.458034:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.458036:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.458037:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.458038:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.458040:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.458042:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.458043:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.458045:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.458046:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.458048:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.458050:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.458051:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.458052:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.458054:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.458056:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.458058:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.458059:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.458061:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.458063:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.458064:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.458066:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.458069:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.458072:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.458074:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.458077:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.458079:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.458080:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.458081:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.458082:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.458084:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.458094:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.458097:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.458098:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.458100:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.458102:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.458104:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.458105:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.458107:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.458108:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.458110:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.458112:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.458113:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.458115:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.458116:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.458120:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.458121:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.458123:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.458124:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.458125:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.458127:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.458129:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.458130:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.458132:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.458134:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105504 00010000:00000001:3.0:1496184948.458136:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.458139:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8d450 x1568861104105504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.458146:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.458147:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.458149:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.458153:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.458155:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.458156:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.458157:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.458159:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.458160:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.458161:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.458164:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.458168:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105504, offset 0 00000100:00000001:3.0:1496184948.458185:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.458186:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.458187:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.458189:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.458190:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.458191:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.458192:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.458194:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8d450 x1568861104105504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.458200:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105504:12345-10.37.248.124@o2ib1:8 Request procesed in 257us (399us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.458206:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6097 00000100:00000040:3.0:1496184948.458208:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.458209:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.458211:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.458214:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132f500. 00000020:00000010:3.0:1496184948.458216:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184948.458218:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000040:3.0:1496184948.458228:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.458230:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.458233:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.458238:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.458239:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.458589:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.458591:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.458593:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23f8c00. 02000000:00000001:7.0:1496184948.458596:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.458597:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00004000:00000001:6.0:1496184948.458970:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.458973:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.458987:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.458988:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.458989:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.458990:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.458993:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.458995:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.458997:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.458998:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.458999:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.459000:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.459002:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.459003:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.459006:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.459007:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.459009:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.459011:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684500. 00000020:00000010:6.0:1496184948.459013:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.459015:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.459016:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.459017:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.459018:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.459019:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.459020:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.459022:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.459023:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.459024:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.459025:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.459026:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.459027:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.459028:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.459029:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.459030:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.459031:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459032:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.459033:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459034:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459035:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.459036:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.459037:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.459038:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.459041:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.459048:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.459048:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.459050:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.459051:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.459052:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.459053:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.459055:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.459056:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.459056:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.459057:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00002000:00000001:6.0:1496184948.459058:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.459059:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.459062:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.459066:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.459068:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.459069:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.459070:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.459071:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.459072:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.459073:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.459075:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57643 00000100:00000040:6.0:1496184948.459077:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.459078:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671268432 : -131872038283184 : ffff881026009850) 00000100:00000040:6.0:1496184948.459080:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.459086:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.459086:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.459088:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.459090:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.459091:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.459093:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.459094:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459095:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.459096:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.459097:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.459098:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.459099:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.459100:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.459101:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459102:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.459104:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.459105:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.459106:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.459107:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.459109:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459110:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.459111:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.459113:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.459117:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.459119:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.459120:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.459122:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.459124:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.459125:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.459126:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.459128:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.459130:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.459131:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.459133:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.459134:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.459135:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.459135:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.459136:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.459138:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.459139:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.459140:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.459142:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.459164:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.459169:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459171:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.459173:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.459175:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.459176:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.459177:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.459178:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.459179:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d600. 00000100:00000010:6.0:1496184948.459181:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.459182:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.459187:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.459188:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.459214:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe830-0x592def0cfe830 00000100:00000001:6.0:1496184948.459216:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.469931:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.469933:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184948.480918:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.480923:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.480923:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184948.491902:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.491966:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.491985:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.491988:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.491991:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.491992:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.491995:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.491996:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.492029:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.492032:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.492034:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.492034:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.492035:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.492036:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.492038:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.492039:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.492040:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.492040:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.492042:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.492043:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.492045:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d600. 00000100:00000001:6.0:1496184948.492046:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.492047:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.492048:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.492050:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.492051:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.492053:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.492056:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.492061:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.492062:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.492063:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.492064:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.492065:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.492067:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.492073:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32986us (33158us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.492078:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57643 00000100:00000040:6.0:1496184948.492080:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.492081:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.492082:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.492085:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.492090:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.492091:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.492092:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.492093:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.492094:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.492096:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.492097:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.492098:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.492099:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.492099:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00002000:00000001:6.0:1496184948.492101:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.492102:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.492103:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684500. 00000020:00000010:6.0:1496184948.492105:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.492107:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.492108:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.492110:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.492112:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.492117:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.492118:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:1.0:1496184948.502749:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184948.502751:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184948.502754:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8dc50 x1568861104105536 msgsize 520 00000100:00100000:1.0:1496184948.502760:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.502762:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.502765:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.502796:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.502799:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.502802:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184948.502804:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.502806:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184948.502807:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184948.502812:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184948.502814:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105536 00000020:00000001:3.0:1496184948.502817:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184948.502818:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184948.502820:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184948.502822:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:3.0:1496184948.502823:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184948.502825:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184948.502829:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.502830:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184948.502832:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000010:3.0:1496184948.502835:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88101132f700. 00000020:00000010:3.0:1496184948.502837:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184948.502839:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.502840:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184948.502842:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184948.502846:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184948.502850:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8dc50 x1568861104105536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184948.502855:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184948.502856:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.502862:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184948.502865:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.502866:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184948.502869:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.502870:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.502871:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184948.502872:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184948.502875:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6098 00000100:00000040:3.0:1496184948.502877:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.502878:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736403536 : -131906973148080 : ffff880803b8dc50) 00000100:00000040:3.0:1496184948.502881:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8dc50 x1568861104105536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184948.502887:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.502887:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184948.502900:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105536:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184948.502902:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105536 00000020:00000001:3.0:1496184948.502904:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184948.502908:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184948.502914:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184948.502918:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184948.502919:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184948.502921:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fb400. 02000000:00000001:3.0:1496184948.502923:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.502924:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.502928:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184948.502930:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184948.502932:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.502934:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184948.502936:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184948.502938:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.502942:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.502943:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.502945:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.502946:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.502948:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.502949:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.502951:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.502953:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.502955:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.502956:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.502957:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.502958:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.502959:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.502964:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.502965:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.502966:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.502968:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.502969:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.502972:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.502973:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.502974:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.502975:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.502977:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.502979:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.502980:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.502981:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.502983:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.502985:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.502986:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.502987:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.502988:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.502990:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.502992:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.502993:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.502994:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.502996:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.502998:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.502999:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.503001:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.503004:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.503006:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.503008:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.503010:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.503012:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.503013:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.503014:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.503015:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.503016:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.503018:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.503020:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.503021:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.503023:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.503024:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.503025:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.503026:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.503028:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.503029:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.503030:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.503032:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.503033:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.503034:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.503036:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.503038:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.503040:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.503042:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.503043:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.503044:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.503045:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.503046:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.503048:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.503050:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.503051:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105536 00010000:00000001:3.0:1496184948.503053:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.503056:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8dc50 x1568861104105536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.503061:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.503062:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.503064:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.503066:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.503068:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.503069:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.503071:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.503072:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.503073:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.503074:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.503076:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.503079:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105536, offset 0 00000100:00000001:3.0:1496184948.503097:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.503099:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.503100:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.503101:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.503102:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.503103:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.503104:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.503106:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8dc50 x1568861104105536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.503112:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105536:12345-10.37.248.124@o2ib1:8 Request procesed in 224us (356us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.503116:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6098 00000100:00000040:3.0:1496184948.503118:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.503120:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.503121:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.503124:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88101132f700. 00000020:00000010:3.0:1496184948.503125:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184948.503127:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88104aaf7000. 00000020:00000040:3.0:1496184948.503129:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.503131:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.503133:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.503138:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.503138:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.503151:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.503153:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.503155:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fb400. 02000000:00000001:7.0:1496184948.503158:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.503159:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.503600:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.503602:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.503604:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ac50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.503610:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.503611:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.503613:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.503645:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.503648:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.503650:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.503651:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.503652:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.503653:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.503656:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.503658:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.503660:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.503661:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.503662:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.503663:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.503665:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.503666:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.503669:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.503670:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.503672:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.503674:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684980. 00000020:00000010:6.0:1496184948.503676:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.503678:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.503679:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.503680:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.503681:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.503682:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.503683:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.503685:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.503686:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.503688:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.503688:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.503690:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.503691:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.503692:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.503692:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.503693:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.503694:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503695:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.503696:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503697:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503698:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.503699:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.503700:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.503702:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.503704:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.503711:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.503712:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.503713:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.503714:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.503715:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.503717:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.503718:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.503719:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.503719:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.503720:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00002000:00000001:6.0:1496184948.503721:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.503722:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.503725:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.503729:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.503732:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.503733:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.503734:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.503735:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.503736:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.503737:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.503739:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57644 00000100:00000040:6.0:1496184948.503741:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.503742:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671273552 : -131872038278064 : ffff88102600ac50) 00000100:00000040:6.0:1496184948.503745:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.503750:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.503750:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.503752:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.503754:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.503755:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.503757:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.503758:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503759:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.503760:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.503761:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.503762:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.503764:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.503764:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.503765:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503766:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.503768:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.503769:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.503770:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.503772:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.503773:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503774:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.503775:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.503777:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.503781:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.503784:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.503785:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.503787:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.503789:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.503790:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.503791:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.503793:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.503795:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.503795:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.503798:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.503799:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.503800:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.503800:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.503801:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.503803:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.503804:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.503805:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.503807:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.503829:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.503835:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503836:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.503839:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.503840:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.503841:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.503842:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.503843:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.503844:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d600. 00000100:00000010:6.0:1496184948.503846:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.503847:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.503851:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.503853:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.503883:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe850-0x592def0cfe850 00000100:00000001:6.0:1496184948.503885:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.505570:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.505571:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184948.505578:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.505582:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.505583:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d600 00000100:00000001:3.0:1496184948.505588:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.505624:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.505631:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.505633:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.505637:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.505638:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.505640:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.505642:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.505675:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.505677:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.505679:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.505680:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.505680:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.505682:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.505683:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.505684:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.505685:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.505685:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.505687:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.505688:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.505690:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d600. 00000100:00000001:6.0:1496184948.505691:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.505692:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.505693:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.505695:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.505696:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.505698:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.505701:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.505706:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.505707:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.505708:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.505709:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.505710:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.505712:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.505718:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1967us (2112us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.505723:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57644 00000100:00000040:6.0:1496184948.505725:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.505726:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.505727:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.505730:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.505735:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.505736:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.505738:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.505739:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.505740:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.505741:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.505743:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.505743:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.505744:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.505745:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00002000:00000001:6.0:1496184948.505746:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.505747:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.505749:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684980. 00000020:00000010:6.0:1496184948.505750:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.505752:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.505753:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.505755:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.505757:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.505762:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.505763:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.534177:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.534179:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.534181:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.534182:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.534183:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.534184:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.534187:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.534188:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105568 00000020:00000001:6.0:1496184948.534190:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.534191:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.534192:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.534193:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.534194:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.534196:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.534198:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.534199:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.534201:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.534203:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684f00. 00000020:00000010:6.0:1496184948.534204:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.534206:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.534207:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.534208:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.534210:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.534213:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c5450 x1568861104105568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.534218:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.534219:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534223:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.534225:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.534226:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.534227:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.534228:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.534229:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.534230:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.534232:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43385 00000100:00000040:6.0:1496184948.534234:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.534235:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676231760 : -131872033319856 : ffff8810264c5450) 00000100:00000040:6.0:1496184948.534237:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5450 x1568861104105568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.534242:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.534243:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.534245:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105568:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.534247:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105568 00000020:00000001:6.0:1496184948.534248:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.534250:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.534253:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.534256:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.534257:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.534258:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a000. 02000000:00000001:6.0:1496184948.534259:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534261:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.534263:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.534265:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.534267:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.534268:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.534270:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.534272:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.534275:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.534276:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.534277:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.534278:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.534279:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.534280:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.534282:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.534283:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.534285:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.534286:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.534287:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.534288:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.534289:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.534292:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.534293:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.534295:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.534297:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.534297:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.534299:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.534301:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.534302:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.534303:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.534305:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.534306:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.534308:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.534309:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.534310:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.534312:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.534313:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.534314:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.534315:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.534317:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.534318:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.534319:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.534320:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.534322:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.534323:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.534325:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534326:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.534329:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.534331:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.534333:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.534335:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.534336:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.534337:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534338:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.534340:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.534341:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.534342:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.534344:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.534345:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534346:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.534347:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.534348:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.534349:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.534351:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.534352:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.534353:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.534355:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.534356:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.534357:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.534359:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.534360:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.534361:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.534363:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.534364:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.534365:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.534366:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.534366:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.534368:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.534369:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.534370:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105568 00010000:00000001:6.0:1496184948.534372:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.534373:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c5450 x1568861104105568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.534379:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.534380:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.534381:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.534383:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.534385:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.534386:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.534387:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.534389:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.534390:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534391:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.534392:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.534395:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105568, offset 0 00000100:00000001:6.0:1496184948.534405:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.534406:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.534408:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.534409:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.534410:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.534410:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.534412:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.534413:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5450 x1568861104105568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.534419:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105568:12345-10.37.248.124@o2ib1:8 Request procesed in 175us (282us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.534423:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43385 00000100:00000040:6.0:1496184948.534425:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.534426:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.534427:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.534428:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684f00. 00000020:00000010:6.0:1496184948.534430:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.534431:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.534433:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.534434:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.534436:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.534439:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.534440:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.535007:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.535010:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.535011:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.535012:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.535013:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.535014:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.535017:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.535019:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.535020:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.535021:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.535022:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.535023:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.535025:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.535026:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.535029:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.535029:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.535031:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.535033:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684680. 00000020:00000010:6.0:1496184948.535034:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.535037:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.535038:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.535039:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.535040:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.535041:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.535042:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.535044:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.535045:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.535046:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.535047:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.535048:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.535049:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.535050:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.535051:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.535052:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.535052:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535054:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.535055:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535056:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535057:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.535058:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.535059:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.535060:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.535063:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.535070:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.535070:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.535072:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.535073:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.535074:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.535075:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.535077:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.535077:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.535078:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.535079:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00002000:00000001:6.0:1496184948.535080:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.535081:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.535083:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.535088:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.535090:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.535091:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.535092:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.535092:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.535093:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.535094:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.535097:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57645 00000100:00000040:6.0:1496184948.535098:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.535099:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671276624 : -131872038274992 : ffff88102600b850) 00000100:00000040:6.0:1496184948.535102:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.535107:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.535108:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.535110:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.535112:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.535113:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.535115:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.535115:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535116:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.535117:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.535119:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.535120:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.535121:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.535121:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.535122:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535123:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.535125:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.535126:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.535127:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e778400. 02000000:00000001:6.0:1496184948.535128:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.535129:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535131:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.535131:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.535133:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.535138:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.535140:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.535141:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.535143:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.535145:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.535146:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.535147:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.535148:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.535150:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.535151:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.535153:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.535154:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.535155:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.535155:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.535156:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.535158:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.535159:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.535160:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.535162:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.535185:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.535191:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535192:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.535194:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.535196:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.535197:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.535198:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.535199:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.535200:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7f200. 00000100:00000010:6.0:1496184948.535202:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.535203:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.535207:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.535208:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.535235:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe870-0x592def0cfe870 00000100:00000001:6.0:1496184948.535237:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.536839:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.536842:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7f200 00000100:00000001:3.0:1496184948.536851:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.536857:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.536858:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7f200 00000100:00000001:3.0:1496184948.536876:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.536916:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.536922:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.536924:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.536928:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.536929:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.536931:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.536933:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.536966:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.536968:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.536970:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.536971:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.536972:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.536974:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.536975:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.536976:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.536977:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.536978:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.536979:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.536980:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.536982:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7f200. 00000100:00000001:6.0:1496184948.536983:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.536984:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.536985:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e778400. 02000000:00000001:6.0:1496184948.536987:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.536988:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.536990:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.536993:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.536998:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.536999:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.537000:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.537000:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.537002:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.537003:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.537009:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1901us (2038us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.537014:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57645 00000100:00000040:6.0:1496184948.537016:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.537017:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.537018:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.537021:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.537026:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.537027:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.537029:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.537030:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.537031:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.537032:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.537034:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.537034:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.537035:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.537036:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00002000:00000001:6.0:1496184948.537037:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.537038:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.537040:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684680. 00000020:00000010:6.0:1496184948.537041:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.537043:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.537044:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.537046:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.537048:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.537053:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.537054:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.565177:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.565179:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.565181:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4850 x1568861104105600 msgsize 520 00000100:00100000:7.0:1496184948.565186:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.565187:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.565189:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.565220:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.565223:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.565225:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.565226:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.565227:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.565228:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.565230:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.565232:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105600 00000020:00000001:6.0:1496184948.565234:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.565235:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.565236:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.565237:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.565238:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.565240:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.565242:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.565243:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.565245:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.565247:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684580. 00000020:00000010:6.0:1496184948.565249:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.565250:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.565251:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.565252:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.565254:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.565257:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c4850 x1568861104105600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.565262:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.565263:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565267:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.565269:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.565270:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.565271:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.565272:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.565273:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.565273:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.565276:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43386 00000100:00000040:6.0:1496184948.565277:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.565278:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676228688 : -131872033322928 : ffff8810264c4850) 00000100:00000040:6.0:1496184948.565281:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4850 x1568861104105600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.565286:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.565287:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.565288:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105600:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.565291:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105600 00000020:00000001:6.0:1496184948.565292:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.565294:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.565297:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.565299:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.565301:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.565302:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778400. 02000000:00000001:6.0:1496184948.565303:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565304:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.565307:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.565308:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.565310:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.565312:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.565313:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.565315:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.565317:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.565319:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.565320:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.565321:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.565321:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.565322:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.565323:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.565325:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.565326:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.565327:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.565328:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.565329:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.565331:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.565334:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.565334:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.565336:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.565338:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.565338:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.565340:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.565342:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.565342:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.565344:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.565345:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.565347:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.565348:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.565349:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.565350:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.565352:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.565353:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.565354:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.565355:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.565357:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.565358:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.565359:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.565360:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.565361:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.565363:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.565364:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565365:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.565367:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.565370:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.565371:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.565373:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.565374:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.565375:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565376:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.565377:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.565378:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.565380:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.565381:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.565382:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565383:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.565384:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.565385:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.565386:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.565388:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.565388:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.565390:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.565391:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.565392:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.565393:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.565395:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.565396:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.565397:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.565399:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.565400:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.565401:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.565401:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.565402:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.565403:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.565405:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.565406:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105600 00010000:00000001:6.0:1496184948.565407:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.565409:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c4850 x1568861104105600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.565414:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.565415:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.565416:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.565419:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.565420:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.565422:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.565423:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.565424:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.565425:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565426:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.565427:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.565430:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105600, offset 0 00000100:00000001:6.0:1496184948.565441:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.565442:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.565443:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.565444:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.565445:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.565446:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.565447:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.565449:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4850 x1568861104105600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.565454:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105600:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (272us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.565459:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43386 00000100:00000040:6.0:1496184948.565460:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.565461:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.565462:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.565464:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684580. 00000020:00000010:6.0:1496184948.565465:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.565467:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.565468:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.565470:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.565472:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.565475:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.565476:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.565536:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.565538:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.565540:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e778400. 02000000:00000001:7.0:1496184948.565543:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.565543:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.565983:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.565985:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.565987:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.565992:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.565993:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.565995:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.566027:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.566029:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.566031:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.566032:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.566033:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.566034:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.566036:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.566038:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.566040:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.566040:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.566042:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.566043:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.566044:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.566046:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.566048:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.566049:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.566051:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.566052:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5580. 00000020:00000010:6.0:1496184948.566054:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.566056:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.566057:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.566058:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.566059:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.566060:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.566061:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.566062:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.566063:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.566065:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.566066:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.566067:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.566068:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.566069:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.566070:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.566071:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.566071:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566072:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.566073:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566074:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566075:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.566077:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.566078:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.566079:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.566082:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.566088:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.566089:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.566091:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.566091:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.566092:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.566094:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.566095:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.566096:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.566097:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.566097:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00002000:00000001:6.0:1496184948.566098:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.566099:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.566102:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.566106:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.566108:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.566109:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.566110:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.566111:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.566112:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.566112:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.566115:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57646 00000100:00000040:6.0:1496184948.566116:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.566118:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671275600 : -131872038276016 : ffff88102600b450) 00000100:00000040:6.0:1496184948.566120:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.566125:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.566126:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.566127:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.566130:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.566131:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.566132:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.566133:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566134:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.566135:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.566136:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.566137:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.566138:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.566139:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.566140:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566141:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.566142:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.566143:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.566145:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77b000. 02000000:00000001:6.0:1496184948.566146:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.566147:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566148:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.566149:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.566151:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.566155:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.566158:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.566158:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.566161:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.566162:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.566163:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.566164:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.566166:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.566167:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.566168:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.566170:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.566172:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.566172:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.566173:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.566174:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.566176:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.566176:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.566177:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.566180:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.566202:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.566208:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566209:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.566212:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.566213:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.566214:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.566215:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.566216:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.566218:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c200. 00000100:00000010:6.0:1496184948.566219:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.566220:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.566225:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.566226:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.566253:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe890-0x592def0cfe890 00000100:00000001:6.0:1496184948.566255:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.567564:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.567567:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.567588:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.567592:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.567592:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.567596:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.567635:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.567642:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.567649:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.567653:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.567654:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.567656:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.567658:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.567691:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.567693:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.567695:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.567696:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.567696:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.567698:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.567699:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.567700:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.567701:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.567702:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.567703:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.567704:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.567706:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c200. 00000100:00000001:6.0:1496184948.567708:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.567709:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.567710:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77b000. 02000000:00000001:6.0:1496184948.567711:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.567713:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.567715:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.567717:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.567722:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.567724:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.567724:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.567725:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.567726:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.567728:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.567734:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1607us (1745us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.567739:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57646 00000100:00000040:6.0:1496184948.567741:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.567742:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.567743:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.567746:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.567751:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.567752:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.567753:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.567754:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.567755:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.567757:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.567758:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.567759:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.567760:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.567760:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00002000:00000001:6.0:1496184948.567762:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.567763:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.567764:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5580. 00000020:00000010:6.0:1496184948.567766:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.567767:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.567769:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.567770:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.567772:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.567778:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.567779:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.596115:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.596117:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.596119:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4c50 x1568861104105632 msgsize 520 00000100:00100000:7.0:1496184948.596125:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.596126:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.596128:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.596161:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.596163:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.596165:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.596166:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.596167:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.596168:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.596170:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.596172:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105632 00000020:00000001:6.0:1496184948.596174:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.596174:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.596176:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.596177:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.596178:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.596180:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.596182:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.596183:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.596184:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.596186:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5c80. 00000020:00000010:6.0:1496184948.596188:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.596189:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.596190:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.596191:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.596193:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.596197:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c4c50 x1568861104105632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.596202:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.596203:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596207:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.596209:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.596210:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.596211:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.596212:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.596213:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.596213:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.596216:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43387 00000100:00000040:6.0:1496184948.596217:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.596219:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676229712 : -131872033321904 : ffff8810264c4c50) 00000100:00000040:6.0:1496184948.596221:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4c50 x1568861104105632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.596226:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.596227:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.596229:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105632:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.596231:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105632 00000020:00000001:6.0:1496184948.596232:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.596234:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.596237:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.596239:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.596240:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.596242:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77b000. 02000000:00000001:6.0:1496184948.596243:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596244:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.596247:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.596248:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.596250:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.596251:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.596253:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.596255:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.596257:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.596259:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.596260:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.596261:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.596262:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.596263:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.596264:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.596265:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.596266:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.596267:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.596268:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.596269:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.596271:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.596274:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.596275:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.596276:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.596278:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.596279:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.596281:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.596282:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.596283:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.596284:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.596286:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.596287:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.596289:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.596290:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.596291:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.596292:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.596294:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.596295:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.596296:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.596297:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.596298:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.596299:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.596300:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.596302:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.596303:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.596304:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596305:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.596308:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.596310:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.596311:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.596313:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.596314:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.596315:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596316:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.596318:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.596319:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.596320:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.596321:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.596323:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596324:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.596325:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.596326:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.596327:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.596328:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.596329:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.596330:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.596332:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.596333:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.596334:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.596335:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.596337:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.596338:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.596339:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.596340:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.596341:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.596342:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.596343:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.596344:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.596345:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.596346:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105632 00010000:00000001:6.0:1496184948.596348:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.596350:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c4c50 x1568861104105632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.596355:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.596356:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.596357:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.596359:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.596361:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.596362:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.596363:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.596365:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.596366:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596367:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.596368:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.596370:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105632, offset 0 00000100:00000001:6.0:1496184948.596381:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.596382:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.596383:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.596385:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.596386:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.596386:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.596387:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.596389:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4c50 x1568861104105632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.596398:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105632:12345-10.37.248.124@o2ib1:8 Request procesed in 171us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.596403:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43387 00000100:00000040:6.0:1496184948.596404:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.596405:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.596406:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.596408:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5c80. 00000020:00000010:6.0:1496184948.596409:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.596411:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.596412:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.596413:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.596415:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.596418:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.596419:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.597024:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.597027:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.597029:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.597030:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.597031:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.597032:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.597035:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.597036:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.597038:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.597039:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.597040:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.597041:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.597043:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.597044:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.597046:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.597047:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.597049:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.597051:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5700. 00000020:00000010:6.0:1496184948.597052:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.597054:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.597056:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.597056:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.597057:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.597059:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.597060:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.597061:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.597062:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.597063:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.597064:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.597065:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.597066:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.597067:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.597068:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.597069:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.597070:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597071:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.597072:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597073:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597074:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.597075:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.597076:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.597077:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.597080:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.597086:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.597087:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.597089:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.597090:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.597091:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.597092:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.597094:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.597094:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.597095:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.597095:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00002000:00000001:6.0:1496184948.597097:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.597098:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.597100:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.597105:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.597107:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.597107:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.597108:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.597109:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.597110:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.597111:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.597113:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57647 00000100:00000040:6.0:1496184948.597115:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.597116:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671278672 : -131872038272944 : ffff88102600c050) 00000100:00000040:6.0:1496184948.597118:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.597123:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.597124:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.597126:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.597128:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.597129:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.597131:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.597132:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597133:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.597133:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.597135:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.597136:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.597137:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.597138:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.597138:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597139:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.597141:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.597142:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.597143:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77e000. 02000000:00000001:6.0:1496184948.597145:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.597146:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597147:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.597148:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.597150:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.597154:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.597156:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.597157:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.597159:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.597161:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.597162:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.597163:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.597164:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.597166:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.597167:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.597169:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.597171:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.597171:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.597172:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.597173:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.597175:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.597175:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.597176:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.597179:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.597201:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.597207:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597208:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.597211:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.597212:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.597213:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.597214:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.597215:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.597216:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7fe00. 00000100:00000010:6.0:1496184948.597218:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.597219:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.597224:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.597225:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.597251:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe8b0-0x592def0cfe8b0 00000100:00000001:6.0:1496184948.597254:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.599048:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.599050:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00000001:3.0:1496184948.610171:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.610176:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.610177:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000001:2.0:1496184948.613911:0:13688:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1496184948.613913:0:13688:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184948.613920:0:13688:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184948.613921:0:13688:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.613923:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184948.613926:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.613927:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184948.613928:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.621176:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.621250:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.621256:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.621258:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.621262:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.621263:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.621265:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.621267:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.621300:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.621302:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.621304:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.621305:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.621306:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.621307:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.621308:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.621309:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.621310:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.621311:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.621312:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.621314:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.621315:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7fe00. 00000100:00000001:6.0:1496184948.621317:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.621318:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.621319:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77e000. 02000000:00000001:6.0:1496184948.621320:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.621322:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.621323:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.621326:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.621331:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.621332:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.621333:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.621334:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.621335:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.621337:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.621343:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24218us (24359us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.621348:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57647 00000100:00000040:6.0:1496184948.621349:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.621351:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.621352:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.621354:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.621360:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.621360:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.621362:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.621363:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.621364:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.621366:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.621367:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.621368:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.621369:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.621369:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00002000:00000001:6.0:1496184948.621370:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.621372:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.621373:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5700. 00000020:00000010:6.0:1496184948.621375:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.621376:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.621378:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.621379:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.621382:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.621387:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.621388:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.631981:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.631983:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.631985:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.631986:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.631987:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.631988:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.631991:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.631992:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105664 00000020:00000001:6.0:1496184948.631994:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.631995:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.631996:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.631997:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.631999:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.632000:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.632002:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.632003:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.632005:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.632007:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5d80. 00000020:00000010:6.0:1496184948.632008:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.632010:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.632011:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.632011:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.632014:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.632017:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c4050 x1568861104105664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.632022:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.632022:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632027:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.632029:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.632030:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.632031:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.632032:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.632033:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.632033:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.632036:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43388 00000100:00000040:6.0:1496184948.632037:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.632038:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676226640 : -131872033324976 : ffff8810264c4050) 00000100:00000040:6.0:1496184948.632041:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4050 x1568861104105664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.632046:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.632047:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.632048:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105664:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.632050:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105664 00000020:00000001:6.0:1496184948.632052:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.632054:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.632057:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.632059:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.632060:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.632061:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77e000. 02000000:00000001:6.0:1496184948.632063:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632064:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.632066:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.632068:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.632070:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.632071:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.632073:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.632075:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.632077:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.632078:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.632079:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.632080:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.632081:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.632082:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.632083:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.632085:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.632086:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.632087:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.632088:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.632089:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.632090:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.632093:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.632094:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.632095:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.632097:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.632098:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.632100:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.632102:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.632102:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.632103:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.632105:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.632107:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.632108:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632109:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.632110:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.632112:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.632113:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632114:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.632115:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.632116:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.632118:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.632119:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.632120:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.632121:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.632122:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.632124:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632125:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.632127:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.632129:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.632131:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.632133:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.632134:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.632135:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632136:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.632137:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.632138:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.632140:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.632141:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.632142:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632143:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.632144:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.632145:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.632146:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.632148:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.632148:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.632149:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.632151:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.632152:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.632153:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.632155:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632156:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.632157:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.632159:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.632159:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.632161:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.632161:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.632162:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632163:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.632165:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.632166:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105664 00010000:00000001:6.0:1496184948.632167:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.632169:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c4050 x1568861104105664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.632174:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.632175:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.632176:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.632178:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.632180:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.632181:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.632183:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.632184:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.632185:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632186:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.632187:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.632189:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105664, offset 0 00000100:00000001:6.0:1496184948.632200:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632201:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.632202:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.632204:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.632204:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.632205:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632206:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.632208:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4050 x1568861104105664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.632214:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105664:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (271us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.632218:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43388 00000100:00000040:6.0:1496184948.632220:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.632221:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.632222:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.632223:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5d80. 00000020:00000010:6.0:1496184948.632225:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.632227:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.632228:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.632229:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.632231:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.632234:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.632235:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.632287:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.632289:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.632291:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77e000. 02000000:00000001:7.0:1496184948.632294:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.632294:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.632728:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.632730:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.632732:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.632737:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.632744:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.632746:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.632780:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.632782:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.632784:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.632785:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.632786:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.632787:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.632789:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.632791:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.632793:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.632793:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.632795:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.632796:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.632797:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.632799:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.632801:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.632802:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.632803:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.632805:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5c00. 00000020:00000010:6.0:1496184948.632807:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.632810:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.632811:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.632812:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.632813:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.632814:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.632815:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.632816:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.632817:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.632819:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.632820:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.632821:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.632822:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.632823:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.632824:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.632825:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.632825:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632826:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.632828:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632829:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632830:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632831:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.632832:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632833:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.632836:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.632842:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.632843:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.632845:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.632845:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.632846:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.632848:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.632849:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.632850:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.632851:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.632851:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00002000:00000001:6.0:1496184948.632853:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632854:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632856:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632860:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.632862:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.632863:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.632864:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.632865:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.632866:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.632867:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.632869:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57648 00000100:00000040:6.0:1496184948.632871:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.632872:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671279696 : -131872038271920 : ffff88102600c450) 00000100:00000040:6.0:1496184948.632874:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.632879:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.632880:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.632882:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.632884:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.632885:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.632886:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.632887:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632888:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.632889:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.632905:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.632906:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.632907:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.632908:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.632908:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632909:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.632911:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.632912:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.632914:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.632915:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632916:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632917:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.632918:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.632920:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.632924:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.632927:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.632928:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.632930:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.632932:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.632933:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.632934:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.632935:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.632937:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.632937:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.632940:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.632941:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.632942:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.632942:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.632943:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.632945:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.632946:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.632947:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.632949:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.632972:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.632978:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632979:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.632982:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632983:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.632984:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.632985:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.632986:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.632987:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d400. 00000100:00000010:6.0:1496184948.632989:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.632990:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.632995:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.632996:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.633022:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe8d0-0x592def0cfe8d0 00000100:00000001:6.0:1496184948.633024:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.634538:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.634541:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.645601:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.645606:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.645607:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.656642:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.656707:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.656725:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.656727:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000001:1.0:1496184948.656730:0:21159:0:(events.c:294:request_in_callback()) Process entered 00002000:00000001:6.0:1496184948.656731:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.656732:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000200:1.0:1496184948.656733:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:6.0:1496184948.656734:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.656736:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000040:1.0:1496184948.656736:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8bc50 x1568861104105696 msgsize 520 00000100:00100000:1.0:1496184948.656743:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184948.656744:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184948.656750:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184948.656756:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.656759:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184948.656763:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184948.656765:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184948.656767:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000001:00000001:6.0:1496184948.656769:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.656769:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.656771:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.656773:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 02000000:00000001:3.0:1496184948.656773:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.656774:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.656775:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.656776:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.656777:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1496184948.656777:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105696 00000020:00000001:6.0:1496184948.656778:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.656779:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000020:00000001:3.0:1496184948.656779:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000100:00000001:6.0:1496184948.656780:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.656781:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.656781:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000100:00000010:6.0:1496184948.656782:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000001:3.0:1496184948.656783:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000010:6.0:1496184948.656784:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d400. 00000100:00000001:6.0:1496184948.656785:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.656786:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 00000020:00000040:3.0:1496184948.656786:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 02000000:00000010:6.0:1496184948.656788:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77dc00. 00000020:00000001:3.0:1496184948.656788:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 02000000:00000001:6.0:1496184948.656789:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:00000001:3.0:1496184948.656789:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:02000400:6.0:1496184948.656791:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.656792:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184948.656794:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00010000:00000080:6.0:1496184948.656795:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.656795:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184948.656798:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88101021e200. 00010000:00000040:6.0:1496184948.656800:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.656802:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.656802:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000010:3.0:1496184948.656802:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100e974f00. 00000020:00000001:6.0:1496184948.656803:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.656804:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000020:00000010:3.0:1496184948.656805:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000040:6.0:1496184948.656806:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.656808:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184948.656809:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184948.656810:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184948.656815:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:6.0:1496184948.656816:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23935us (24082us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.656821:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57648 00000100:00000200:3.0:1496184948.656821:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8bc50 x1568861104105696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000040:6.0:1496184948.656823:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.656825:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.656825:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000001:3.0:1496184948.656827:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00002000:00100000:6.0:1496184948.656828:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184948.656828:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.656832:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00002000:00000001:6.0:1496184948.656834:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.656834:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184948.656835:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00010000:00000001:6.0:1496184948.656836:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.656837:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00004000:00000001:3.0:1496184948.656837:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:00000001:6.0:1496184948.656838:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00004000:00000001:3.0:1496184948.656838:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184948.656839:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00010000:00000001:6.0:1496184948.656840:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000100:00000001:3.0:1496184948.656840:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00010000:00000001:6.0:1496184948.656841:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.656842:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.656842:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00000100:00000001:3.0:1496184948.656842:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00002000:00010000:6.0:1496184948.656843:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00002000:00000001:6.0:1496184948.656844:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184948.656845:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6099 00000100:00000001:6.0:1496184948.656846:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.656847:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5c00. 00000100:00000040:3.0:1496184948.656847:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000020:00000010:6.0:1496184948.656849:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000100:00000001:3.0:1496184948.656850:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736395344 : -131906973156272 : ffff880803b8bc50) 00000020:00000010:6.0:1496184948.656851:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.656852:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000040:3.0:1496184948.656853:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8bc50 x1568861104105696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.656854:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.656856:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00000001:3.0:1496184948.656858:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184948.656859:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00004000:00000001:6.0:1496184948.656861:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000100:00100000:3.0:1496184948.656861:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105696:12345-10.37.248.124@o2ib1:8 00004000:00000001:6.0:1496184948.656863:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000200:3.0:1496184948.656864:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105696 00000020:00000001:3.0:1496184948.656866:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184948.656870:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184948.656878:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184948.656883:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184948.656885:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184948.656887:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fe000. 02000000:00000001:3.0:1496184948.656889:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.656903:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.656909:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184948.656911:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184948.656914:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184948.656916:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184948.656918:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184948.656920:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:3.0:1496184948.656924:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184948.656927:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184948.656929:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184948.656931:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184948.656933:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184948.656935:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184948.656937:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184948.656938:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184948.656940:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184948.656942:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184948.656943:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184948.656946:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184948.656947:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184948.656952:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184948.656954:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184948.656955:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184948.656958:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184948.656959:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184948.656962:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184948.656964:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184948.656965:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184948.656966:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184948.656968:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184948.656970:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184948.656972:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.656973:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184948.656975:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184948.656977:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184948.656978:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184948.656980:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.656981:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184948.656983:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184948.656985:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.656987:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184948.656987:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184948.656990:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.656992:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.656993:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.656995:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184948.656998:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184948.657001:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184948.657004:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184948.657006:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.657008:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.657009:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.657010:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.657012:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.657013:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.657015:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184948.657017:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184948.657019:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.657021:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184948.657023:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184948.657024:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184948.657025:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184948.657027:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184948.657028:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184948.657030:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184948.657032:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184948.657033:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184948.657035:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184948.657037:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.657040:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184948.657042:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184948.657044:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184948.657045:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184948.657046:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184948.657048:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184948.657049:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.657051:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184948.657053:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184948.657055:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105696 00010000:00000001:3.0:1496184948.657057:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184948.657060:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8bc50 x1568861104105696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184948.657067:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184948.657068:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184948.657070:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184948.657082:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184948.657084:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184948.657085:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.657086:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184948.657088:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184948.657089:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.657090:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184948.657092:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184948.657097:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105696, offset 0 00000100:00000001:3.0:1496184948.657113:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.657115:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184948.657116:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184948.657117:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184948.657118:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184948.657119:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184948.657120:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184948.657122:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8bc50 x1568861104105696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184948.657128:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105696:12345-10.37.248.124@o2ib1:8 Request procesed in 269us (391us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184948.657133:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6099 00000100:00000040:3.0:1496184948.657135:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184948.657137:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184948.657138:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184948.657141:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100e974f00. 00000020:00000010:3.0:1496184948.657144:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184948.657146:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88101021e200. 00000020:00000040:3.0:1496184948.657147:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184948.657149:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184948.657153:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184948.657166:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184948.657167:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.657475:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.657478:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.657479:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fe000. 02000000:00000001:7.0:1496184948.657482:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.657483:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.657932:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.657934:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.657937:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.657942:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.657943:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.657945:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.657992:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.657996:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.657999:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.658001:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.658003:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.658004:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.658008:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.658011:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.658014:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.658015:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.658017:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.658019:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.658021:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.658024:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.658027:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.658028:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.658031:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.658045:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5000. 00000020:00000010:6.0:1496184948.658047:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.658049:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.658050:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.658051:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.658052:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.658053:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.658055:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.658056:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.658057:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.658058:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.658059:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.658060:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.658061:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.658063:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.658063:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.658064:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.658065:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658066:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.658067:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658068:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658069:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.658070:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.658071:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.658072:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.658075:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.658082:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.658082:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.658084:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.658085:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.658086:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.658087:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.658089:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.658090:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.658090:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.658091:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00002000:00000001:6.0:1496184948.658092:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.658093:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.658095:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.658100:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.658102:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.658103:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.658104:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.658105:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.658106:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.658106:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.658109:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57649 00000100:00000040:6.0:1496184948.658111:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.658112:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671277648 : -131872038273968 : ffff88102600bc50) 00000100:00000040:6.0:1496184948.658114:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.658120:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.658120:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.658122:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.658124:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.658125:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.658127:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.658128:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658129:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.658130:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.658131:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.658132:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.658133:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.658134:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.658135:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658136:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.658138:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.658139:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.658140:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.658141:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.658143:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658144:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.658145:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.658147:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.658151:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.658154:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.658154:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.658157:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.658158:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.658160:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.658161:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.658162:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.658164:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.658165:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.658167:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.658169:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.658170:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.658170:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.658171:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.658173:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.658174:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.658175:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.658177:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.658200:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.658205:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658207:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.658209:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.658211:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.658212:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.658212:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.658214:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.658215:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d400. 00000100:00000010:6.0:1496184948.658216:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.658218:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.658222:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.658223:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.658249:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe8f0-0x592def0cfe8f0 00000100:00000001:6.0:1496184948.658251:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.670000:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.670002:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.681000:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.681005:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.681006:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.691987:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.692046:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.692065:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.692067:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.692071:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.692072:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.692074:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.692076:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.692108:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.692111:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.692113:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.692113:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.692114:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.692116:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.692117:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.692118:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.692119:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.692120:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.692121:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.692123:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.692124:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d400. 00000100:00000001:6.0:1496184948.692126:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.692126:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.692128:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.692129:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.692131:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.692132:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.692135:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.692140:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.692141:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.692142:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.692143:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.692144:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.692146:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.692152:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34031us (34213us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.692157:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57649 00000100:00000040:6.0:1496184948.692159:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.692160:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.692161:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.692164:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.692174:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.692174:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.692176:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.692177:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.692178:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.692180:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.692181:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.692182:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.692182:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.692183:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00002000:00000001:6.0:1496184948.692184:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.692186:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.692187:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5000. 00000020:00000010:6.0:1496184948.692189:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.692190:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.692192:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.692193:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.692196:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.692201:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.692202:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184948.702640:0:1272:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:3.0:1496184948.702642:0:1272:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:3.0:1496184948.702645:0:1272:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c450 x1568861104105728 msgsize 520 00000100:00100000:3.0:1496184948.702651:0:1272:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:3.0:1496184948.702652:0:1272:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:3.0:1496184948.702655:0:1272:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:0.0:1496184948.702689:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:0.0:1496184948.702692:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:0.0:1496184948.702695:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:0.0:1496184948.702697:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:0.0:1496184948.702698:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:0.0:1496184948.702700:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1496184948.702704:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:0.0:1496184948.702707:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105728 00000020:00000001:0.0:1496184948.702708:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:0.0:1496184948.702709:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:0.0:1496184948.702711:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:0.0:1496184948.702713:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:0.0:1496184948.702715:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:0.0:1496184948.702717:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:0.0:1496184948.702720:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:0.0:1496184948.702721:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:0.0:1496184948.702726:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff880814344400. 00000020:00000010:0.0:1496184948.702729:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808f4f10e80. 00000020:00000010:0.0:1496184948.702732:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88105b797f00. 00000100:00000001:0.0:1496184948.702733:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:0.0:1496184948.702734:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:0.0:1496184948.702736:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184948.702739:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:0.0:1496184948.702743:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8c450 x1568861104105728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184948.702748:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:0.0:1496184948.702749:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702755:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:0.0:1496184948.702757:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:0.0:1496184948.702758:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:0.0:1496184948.702759:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184948.702760:0:14310:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:0.0:1496184948.702760:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:0.0:1496184948.702761:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:0.0:1496184948.702762:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00004000:00000001:1.0:1496184948.702763:0:14310:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184948.702765:0:14310:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00100000:0.0:1496184948.702765:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6100 00000100:00000001:1.0:1496184948.702766:0:14310:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000040:0.0:1496184948.702767:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:1.0:1496184948.702768:0:14310:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702769:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736397392 : -131906973154224 : ffff880803b8c450) 00000100:00000001:1.0:1496184948.702770:0:14310:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1496184948.702771:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c450 x1568861104105728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00004000:00000001:1.0:1496184948.702773:0:14310:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184948.702774:0:14310:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:0.0:1496184948.702777:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:0.0:1496184948.702777:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:0.0:1496184948.702779:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105728:12345-10.37.248.124@o2ib1:8 00000100:00000200:0.0:1496184948.702782:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105728 00000020:00000001:0.0:1496184948.702783:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:0.0:1496184948.702786:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:0.0:1496184948.702791:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:0.0:1496184948.702795:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:0.0:1496184948.702796:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:0.0:1496184948.702798:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807fc3c1800. 02000000:00000001:0.0:1496184948.702800:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702802:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:0.0:1496184948.702806:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:0.0:1496184948.702808:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:0.0:1496184948.702810:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:0.0:1496184948.702812:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:0.0:1496184948.702813:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:0.0:1496184948.702816:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:0.0:1496184948.702818:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:0.0:1496184948.702820:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:0.0:1496184948.702822:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:0.0:1496184948.702823:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:0.0:1496184948.702825:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:0.0:1496184948.702826:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:0.0:1496184948.702827:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:0.0:1496184948.702829:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:0.0:1496184948.702830:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:0.0:1496184948.702831:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:0.0:1496184948.702833:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:0.0:1496184948.702834:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:0.0:1496184948.702835:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:0.0:1496184948.702839:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:0.0:1496184948.702840:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:0.0:1496184948.702841:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:0.0:1496184948.702844:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:0.0:1496184948.702844:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:0.0:1496184948.702847:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:0.0:1496184948.702848:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:0.0:1496184948.702849:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:0.0:1496184948.702850:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:0.0:1496184948.702852:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:0.0:1496184948.702854:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:0.0:1496184948.702856:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:0.0:1496184948.702857:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:0.0:1496184948.702859:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:0.0:1496184948.702860:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:0.0:1496184948.702862:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:0.0:1496184948.702863:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1496184948.702864:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1496184948.702865:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:0.0:1496184948.702867:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184948.702868:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:0.0:1496184948.702869:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:0.0:1496184948.702870:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:0.0:1496184948.702872:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:0.0:1496184948.702873:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702875:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:0.0:1496184948.702878:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:0.0:1496184948.702880:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:0.0:1496184948.702882:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:0.0:1496184948.702884:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:0.0:1496184948.702885:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:0.0:1496184948.702886:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702887:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:0.0:1496184948.702889:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:0.0:1496184948.702902:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:0.0:1496184948.702904:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:0.0:1496184948.702906:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:0.0:1496184948.702908:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702909:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:0.0:1496184948.702911:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:0.0:1496184948.702911:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:0.0:1496184948.702912:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:0.0:1496184948.702914:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:0.0:1496184948.702916:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:0.0:1496184948.702917:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:0.0:1496184948.702919:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:0.0:1496184948.702920:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:0.0:1496184948.702922:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:0.0:1496184948.702923:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1496184948.702925:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:0.0:1496184948.702927:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:0.0:1496184948.702928:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:0.0:1496184948.702929:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:0.0:1496184948.702930:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:0.0:1496184948.702932:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:0.0:1496184948.702933:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1496184948.702934:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:0.0:1496184948.702936:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:0.0:1496184948.702938:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105728 00010000:00000001:0.0:1496184948.702939:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:0.0:1496184948.702942:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8c450 x1568861104105728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:0.0:1496184948.702947:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:0.0:1496184948.702948:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:0.0:1496184948.702950:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:0.0:1496184948.702952:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:0.0:1496184948.702954:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:0.0:1496184948.702955:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:0.0:1496184948.702956:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:0.0:1496184948.702958:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:0.0:1496184948.702959:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702960:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:0.0:1496184948.702962:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:0.0:1496184948.702965:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105728, offset 0 00000100:00000001:0.0:1496184948.702979:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.702980:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:0.0:1496184948.702981:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:0.0:1496184948.702983:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1496184948.702984:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:0.0:1496184948.702986:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1496184948.702987:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:0.0:1496184948.702989:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c450 x1568861104105728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:0.0:1496184948.702994:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105728:12345-10.37.248.124@o2ib1:8 Request procesed in 217us (348us total) trans 0 rc 0/0 00000100:00100000:0.0:1496184948.702999:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6100 00000100:00000040:0.0:1496184948.703001:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:0.0:1496184948.703003:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:0.0:1496184948.703004:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:0.0:1496184948.703006:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808f4f10e80. 00000020:00000010:0.0:1496184948.703008:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88105b797f00. 00000020:00000010:0.0:1496184948.703009:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff880814344400. 00000020:00000040:0.0:1496184948.703011:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:0.0:1496184948.703013:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:0.0:1496184948.703015:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:0.0:1496184948.703019:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:0.0:1496184948.703020:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.703034:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.703036:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.703038:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807fc3c1800. 02000000:00000001:7.0:1496184948.703041:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.703042:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.703477:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.703479:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.703481:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.703487:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.703488:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.703490:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.703528:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.703531:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.703533:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.703534:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.703535:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.703536:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.703538:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.703540:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.703542:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.703543:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.703545:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.703546:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.703547:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.703549:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.703552:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.703553:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.703555:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.703557:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5500. 00000020:00000010:6.0:1496184948.703559:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.703561:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.703562:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.703563:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.703564:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.703565:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.703566:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.703568:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.703569:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.703570:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.703571:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.703572:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.703573:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.703574:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.703575:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.703576:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.703576:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703577:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.703578:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703579:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703581:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.703582:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.703583:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.703584:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.703587:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.703593:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.703594:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.703596:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.703596:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.703597:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.703599:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.703600:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.703601:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.703602:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.703602:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00002000:00000001:6.0:1496184948.703604:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.703605:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.703607:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.703611:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.703614:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.703614:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.703615:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.703616:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.703617:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.703618:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.703621:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57650 00000100:00000040:6.0:1496184948.703622:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.703623:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671282768 : -131872038268848 : ffff88102600d050) 00000100:00000040:6.0:1496184948.703626:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.703631:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.703632:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.703633:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.703635:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.703637:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.703638:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.703639:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703640:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.703641:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.703642:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.703644:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.703645:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.703645:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.703646:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703647:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.703649:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.703650:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.703652:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.703653:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.703654:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703656:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.703656:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.703658:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.703662:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.703665:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.703666:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.703668:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.703670:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.703671:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.703672:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.703673:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.703675:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.703676:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.703678:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.703680:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.703680:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.703681:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.703682:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.703684:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.703684:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.703685:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.703688:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.703710:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.703715:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703717:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.703719:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.703721:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.703722:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.703723:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.703724:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.703725:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7d400. 00000100:00000010:6.0:1496184948.703727:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.703728:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.703732:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.703733:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.703760:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe910-0x592def0cfe910 00000100:00000001:6.0:1496184948.703762:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.705219:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.705222:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.705242:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.705246:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.705247:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7d400 00000100:00000001:3.0:1496184948.705252:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.705291:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.705298:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.705300:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.705304:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.705305:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.705307:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.705309:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.705342:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.705344:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.705346:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.705347:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.705347:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.705349:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.705350:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.705351:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.705352:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.705353:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.705354:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.705355:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.705357:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7d400. 00000100:00000001:6.0:1496184948.705358:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.705359:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.705360:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.705362:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.705364:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.705365:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.705368:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.705373:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.705375:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.705375:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.705376:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.705377:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.705379:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.705385:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1753us (1902us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.705390:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57650 00000100:00000040:6.0:1496184948.705392:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.705393:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.705394:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.705397:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.705402:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.705403:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.705404:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.705405:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.705407:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.705408:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.705409:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.705410:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.705411:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.705412:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00002000:00000001:6.0:1496184948.705413:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.705414:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.705416:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5500. 00000020:00000010:6.0:1496184948.705417:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.705419:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.705420:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.705422:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.705424:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.705429:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.705430:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.733977:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.733979:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.733982:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c4450 x1568861104105760 msgsize 520 00000100:00100000:7.0:1496184948.733987:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.733988:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.733990:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.734020:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.734023:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.734025:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.734026:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.734027:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.734028:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.734030:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.734032:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105760 00000020:00000001:6.0:1496184948.734034:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.734034:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.734036:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.734037:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.734038:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.734040:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.734042:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.734043:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.734044:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.734046:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5300. 00000020:00000010:6.0:1496184948.734048:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.734049:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.734050:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.734051:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.734053:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.734056:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c4450 x1568861104105760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.734065:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.734066:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734070:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.734073:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.734073:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.734074:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.734075:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.734076:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.734076:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.734079:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43389 00000100:00000040:6.0:1496184948.734080:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.734081:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676227664 : -131872033323952 : ffff8810264c4450) 00000100:00000040:6.0:1496184948.734084:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c4450 x1568861104105760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.734089:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.734090:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.734091:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105760:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.734093:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105760 00000020:00000001:6.0:1496184948.734095:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.734097:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.734100:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.734102:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.734103:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.734105:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77dc00. 02000000:00000001:6.0:1496184948.734106:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734107:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.734110:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.734111:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.734113:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.734115:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.734116:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.734118:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.734121:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.734123:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.734124:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.734125:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.734126:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.734127:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.734128:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.734130:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.734131:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.734132:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.734133:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.734135:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.734136:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.734139:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.734140:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.734141:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.734143:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.734144:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.734146:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.734147:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.734148:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.734149:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.734151:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.734153:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.734154:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.734155:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.734156:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.734158:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.734160:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.734161:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.734162:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.734163:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.734165:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.734166:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.734167:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.734168:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.734170:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.734171:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734172:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.734175:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.734177:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.734179:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.734181:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.734182:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.734183:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734184:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.734186:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.734187:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.734188:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.734190:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.734191:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734192:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.734193:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.734194:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.734195:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.734197:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.734198:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.734199:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.734201:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.734202:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.734203:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.734205:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734206:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.734207:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.734209:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.734210:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.734211:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.734212:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.734212:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734214:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.734215:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.734216:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105760 00010000:00000001:6.0:1496184948.734218:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.734219:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c4450 x1568861104105760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.734225:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.734226:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.734227:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.734229:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.734231:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.734232:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.734233:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.734235:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.734236:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734237:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.734238:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.734241:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105760, offset 0 00000100:00000001:6.0:1496184948.734252:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734253:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.734254:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.734255:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.734256:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.734257:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734258:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.734259:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c4450 x1568861104105760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.734265:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105760:12345-10.37.248.124@o2ib1:8 Request procesed in 175us (282us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.734269:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43389 00000100:00000040:6.0:1496184948.734271:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.734272:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.734273:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.734274:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5300. 00000020:00000010:6.0:1496184948.734276:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.734277:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.734279:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.734280:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.734282:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.734285:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.734286:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.734342:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.734344:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.734346:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77dc00. 02000000:00000001:7.0:1496184948.734349:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.734349:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.734761:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.734763:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.734765:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.734770:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.734771:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.734773:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.734803:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.734806:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.734807:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.734808:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.734809:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.734810:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.734813:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.734815:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.734816:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.734817:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.734818:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.734819:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.734821:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.734822:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.734824:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.734825:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.734827:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.734829:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5280. 00000020:00000010:6.0:1496184948.734830:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.734832:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.734834:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.734834:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.734836:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.734837:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.734838:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.734839:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.734840:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.734841:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.734842:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.734843:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.734844:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.734845:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.734846:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.734847:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.734848:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734849:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.734850:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734851:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734852:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.734853:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.734854:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.734855:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.734858:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.734865:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.734865:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.734867:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.734868:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.734869:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.734870:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.734872:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.734872:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.734873:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.734874:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00002000:00000001:6.0:1496184948.734875:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734876:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734878:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734883:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.734885:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.734886:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.734887:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.734887:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.734888:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.734901:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.734903:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57651 00000100:00000040:6.0:1496184948.734905:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.734906:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671280720 : -131872038270896 : ffff88102600c850) 00000100:00000040:6.0:1496184948.734910:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.734915:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.734915:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.734917:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.734919:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.734920:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.734922:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.734923:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734924:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.734925:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.734926:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.734927:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.734928:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.734929:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.734929:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734931:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.734932:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.734933:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.734935:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a800. 02000000:00000001:6.0:1496184948.734936:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.734937:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.734939:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.734939:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.734941:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.734945:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.734948:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.734949:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.734951:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.734953:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.734954:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.734955:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.734956:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.734958:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.734959:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.734961:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.734962:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.734963:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.734963:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.734964:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.734966:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.734967:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.734968:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.734970:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.734993:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.734999:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.735000:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.735003:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.735004:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.735005:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.735006:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.735007:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.735008:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e600. 00000100:00000010:6.0:1496184948.735010:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.735011:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.735016:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.735017:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.735043:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe930-0x592def0cfe930 00000100:00000001:6.0:1496184948.735046:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.736468:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.736470:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184948.736479:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.736496:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.736496:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e600 00000100:00000001:3.0:1496184948.736501:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.736538:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.736544:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.736547:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.736550:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.736551:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.736553:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.736555:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.736588:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.736590:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.736592:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.736593:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.736594:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.736595:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.736596:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.736598:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.736598:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.736599:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.736600:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.736602:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.736604:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e600. 00000100:00000001:6.0:1496184948.736605:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.736606:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.736607:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a800. 02000000:00000001:6.0:1496184948.736608:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.736610:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.736612:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.736614:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.736619:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.736621:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.736622:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.736622:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.736624:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.736625:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.736631:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1715us (1865us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.736636:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57651 00000100:00000040:6.0:1496184948.736638:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.736639:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.736640:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.736643:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.736648:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.736649:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.736651:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.736652:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.736653:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.736654:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.736656:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.736656:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.736657:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.736657:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00002000:00000001:6.0:1496184948.736659:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.736660:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.736661:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5280. 00000020:00000010:6.0:1496184948.736663:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.736665:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.736666:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.736668:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.736670:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.736675:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.736676:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.764475:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.764477:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.764479:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3050 x1568861104105792 msgsize 520 00000100:00100000:7.0:1496184948.764484:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.764485:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.764488:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.764520:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.764522:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.764524:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.764525:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.764526:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.764527:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.764530:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.764531:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105792 00000020:00000001:6.0:1496184948.764533:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.764534:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.764535:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.764536:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.764538:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.764539:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.764541:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.764542:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.764544:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.764546:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5980. 00000020:00000010:6.0:1496184948.764547:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.764549:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.764549:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.764550:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.764552:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.764556:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c3050 x1568861104105792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.764561:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.764561:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764566:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.764568:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.764569:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.764570:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.764570:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.764571:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.764572:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.764574:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43390 00000100:00000040:6.0:1496184948.764576:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.764577:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676222544 : -131872033329072 : ffff8810264c3050) 00000100:00000040:6.0:1496184948.764580:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3050 x1568861104105792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.764585:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.764585:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.764587:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105792:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.764589:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105792 00000020:00000001:6.0:1496184948.764590:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.764592:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.764595:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.764598:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.764599:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.764600:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a800. 02000000:00000001:6.0:1496184948.764602:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764603:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.764605:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.764607:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.764609:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.764610:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.764612:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.764614:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.764616:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.764617:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.764618:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.764619:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.764620:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.764621:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.764622:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.764623:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.764625:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.764626:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.764627:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.764628:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.764629:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.764632:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.764633:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.764635:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.764636:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.764637:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.764639:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.764641:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.764641:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.764642:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.764644:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.764646:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.764647:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.764648:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.764649:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.764651:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.764652:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.764653:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.764654:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.764655:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.764657:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.764658:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.764659:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.764660:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.764661:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.764663:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764664:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.764666:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.764669:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.764671:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.764672:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.764674:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.764675:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764676:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.764677:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.764678:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.764680:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.764681:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.764682:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764683:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.764684:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.764685:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.764686:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.764688:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.764688:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.764689:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.764691:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.764692:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.764693:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.764695:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.764696:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.764697:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.764699:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.764699:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.764700:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.764701:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.764702:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.764703:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.764704:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.764705:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105792 00010000:00000001:6.0:1496184948.764707:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.764709:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c3050 x1568861104105792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.764714:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.764715:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.764716:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.764718:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.764720:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.764721:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.764722:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.764724:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.764725:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764726:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.764727:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.764729:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105792, offset 0 00000100:00000001:6.0:1496184948.764740:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.764741:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.764742:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.764743:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.764744:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.764745:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.764746:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.764748:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3050 x1568861104105792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.764753:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105792:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (273us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.764758:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43390 00000100:00000040:6.0:1496184948.764760:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.764761:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.764762:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.764763:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5980. 00000020:00000010:6.0:1496184948.764765:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.764766:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.764768:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.764769:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.764771:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.764774:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.764775:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.764829:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.764831:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.764832:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77a800. 02000000:00000001:7.0:1496184948.764835:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.764836:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.765308:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.765310:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.765312:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600cc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.765317:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.765318:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.765320:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.765352:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.765354:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.765356:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.765357:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.765358:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.765359:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.765362:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.765363:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.765365:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.765366:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.765367:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.765368:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.765370:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.765371:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.765373:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.765374:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.765376:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.765378:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5380. 00000020:00000010:6.0:1496184948.765379:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.765381:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.765383:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.765383:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.765384:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.765385:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.765386:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.765388:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.765389:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.765390:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.765391:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.765392:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.765393:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.765394:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.765395:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.765396:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.765397:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765398:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.765399:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765400:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765401:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.765402:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.765403:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.765404:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.765407:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.765413:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.765414:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.765416:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.765417:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.765418:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.765419:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.765421:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.765421:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.765422:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.765422:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00002000:00000001:6.0:1496184948.765424:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.765425:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.765427:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.765431:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.765433:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.765434:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.765435:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.765436:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.765437:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.765438:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.765440:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57652 00000100:00000040:6.0:1496184948.765442:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.765443:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671281744 : -131872038269872 : ffff88102600cc50) 00000100:00000040:6.0:1496184948.765445:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.765451:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.765451:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.765453:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.765455:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.765456:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.765458:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.765459:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765460:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.765461:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.765462:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.765463:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.765464:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.765465:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.765466:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765467:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.765468:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.765469:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.765471:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77d800. 02000000:00000001:6.0:1496184948.765472:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.765473:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765474:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.765475:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.765477:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.765481:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.765484:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.765485:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.765487:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.765488:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.765489:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.765490:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.765492:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.765494:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.765494:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.765496:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.765498:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.765499:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.765499:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.765500:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.765502:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.765502:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.765503:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.765506:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.765527:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.765533:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765535:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.765537:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.765538:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.765539:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.765540:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.765541:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.765543:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7da00. 00000100:00000010:6.0:1496184948.765544:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.765546:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.765550:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.765551:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.765578:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe950-0x592def0cfe950 00000100:00000001:6.0:1496184948.765580:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184948.767067:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:0.0:1496184948.767081:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7da00 00000100:00000001:0.0:1496184948.767087:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:0.0:1496184948.767091:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:0.0:1496184948.767092:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7da00 00000100:00000001:0.0:1496184948.767097:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.767136:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.767142:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.767145:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.767148:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.767154:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.767156:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.767158:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.767191:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.767193:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.767195:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.767196:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.767196:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.767198:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.767199:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.767200:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.767201:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.767201:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.767202:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.767204:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.767206:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7da00. 00000100:00000001:6.0:1496184948.767207:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.767208:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.767209:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77d800. 02000000:00000001:6.0:1496184948.767210:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.767212:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.767214:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.767216:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.767221:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.767223:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.767224:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.767224:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.767226:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.767227:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.767233:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1781us (1919us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.767238:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57652 00000100:00000040:6.0:1496184948.767240:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.767241:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.767242:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.767245:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.767250:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.767250:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.767252:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.767253:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.767254:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.767256:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.767257:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.767258:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.767259:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.767259:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00002000:00000001:6.0:1496184948.767260:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.767262:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.767263:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5380. 00000020:00000010:6.0:1496184948.767265:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.767266:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.767268:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.767269:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.767271:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.767276:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.767278:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.794981:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.794984:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.794985:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.794986:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.794987:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.794988:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.794991:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.794993:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105824 00000020:00000001:6.0:1496184948.794994:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.794995:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.794997:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.794998:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.794999:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.795001:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.795003:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.795004:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.795005:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.795007:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5100. 00000020:00000010:6.0:1496184948.795009:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.795010:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.795011:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.795012:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.795014:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.795017:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c3c50 x1568861104105824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.795022:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.795023:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795028:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.795030:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.795030:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.795031:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.795032:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.795033:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.795034:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.795036:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43391 00000100:00000040:6.0:1496184948.795037:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.795039:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676225616 : -131872033326000 : ffff8810264c3c50) 00000100:00000040:6.0:1496184948.795041:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3c50 x1568861104105824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.795046:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.795047:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.795049:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105824:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.795052:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105824 00000020:00000001:6.0:1496184948.795053:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.795055:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.795058:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.795061:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.795062:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.795063:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77d800. 02000000:00000001:6.0:1496184948.795064:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795066:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.795068:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.795070:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.795071:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.795073:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.795074:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.795076:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.795079:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.795080:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.795081:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.795082:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.795083:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.795084:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.795085:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.795086:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.795087:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.795088:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.795089:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.795090:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.795092:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.795095:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.795096:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.795097:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.795100:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.795101:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.795103:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.795104:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.795105:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.795106:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.795108:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.795109:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.795110:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.795111:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.795112:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.795114:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.795115:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.795116:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.795117:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.795119:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.795120:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.795121:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.795122:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.795123:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.795125:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.795126:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795127:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.795129:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.795132:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.795133:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.795135:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.795136:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.795137:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795138:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.795139:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.795141:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.795142:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.795143:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.795144:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795146:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.795147:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.795147:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.795148:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.795150:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.795151:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.795152:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.795153:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.795155:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.795156:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.795157:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795159:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.795160:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.795161:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.795162:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.795163:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.795164:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.795164:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795166:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.795167:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.795168:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105824 00010000:00000001:6.0:1496184948.795170:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.795171:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c3c50 x1568861104105824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.795177:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.795178:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.795179:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.795181:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.795183:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.795184:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.795185:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.795187:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.795187:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795188:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.795190:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.795192:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105824, offset 0 00000100:00000001:6.0:1496184948.795203:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795204:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.795205:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.795206:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.795207:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.795208:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795209:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.795211:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3c50 x1568861104105824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.795216:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105824:12345-10.37.248.124@o2ib1:8 Request procesed in 169us (274us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.795221:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43391 00000100:00000040:6.0:1496184948.795222:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.795223:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.795224:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.795226:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5100. 00000020:00000010:6.0:1496184948.795227:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.795229:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.795230:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.795232:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.795233:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.795237:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.795237:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.795848:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.795850:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.795852:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.795853:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.795854:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.795855:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.795858:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.795860:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.795861:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.795862:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.795863:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.795864:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.795866:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.795867:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.795869:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.795870:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.795872:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.795874:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5480. 00000020:00000010:6.0:1496184948.795875:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.795878:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.795879:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.795880:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.795881:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.795882:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.795883:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.795885:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.795886:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.795887:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.795888:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.795889:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.795905:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.795906:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.795907:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.795908:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.795908:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795909:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.795911:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795912:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795913:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.795914:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.795915:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.795916:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.795919:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.795933:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.795934:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.795935:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.795936:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.795937:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.795939:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.795940:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.795941:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.795941:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.795942:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00002000:00000001:6.0:1496184948.795943:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795944:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795947:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795951:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.795953:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.795953:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.795954:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.795955:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.795956:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.795957:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.795959:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57653 00000100:00000040:6.0:1496184948.795961:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.795962:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671283792 : -131872038267824 : ffff88102600d450) 00000100:00000040:6.0:1496184948.795964:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.795970:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.795970:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.795972:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.795974:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.795975:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.795977:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.795978:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795979:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.795979:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.795981:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.795982:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.795983:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.795984:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.795984:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795985:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.795987:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.795988:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.795990:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e779800. 02000000:00000001:6.0:1496184948.795991:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.795992:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.795993:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.795994:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.795996:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.796000:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.796003:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.796003:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.796006:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.796007:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.796008:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.796009:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.796011:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.796012:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.796013:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.796015:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.796017:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.796017:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.796018:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.796019:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.796021:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.796021:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.796022:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.796025:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.796047:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.796052:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.796054:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.796056:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.796058:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.796059:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.796060:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.796061:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.796062:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7e200. 00000100:00000010:6.0:1496184948.796064:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.796065:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.796070:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.796071:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.796096:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe970-0x592def0cfe970 00000100:00000001:6.0:1496184948.796099:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.797620:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.797622:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7e200 00000100:00000001:3.0:1496184948.808749:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.808754:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.808755:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7e200 00000100:00000001:3.0:1496184948.819740:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.819795:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.819814:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.819816:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.819820:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.819821:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.819823:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.819825:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.819858:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.819860:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.819862:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.819863:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.819863:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.819865:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.819866:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.819868:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.819868:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.819869:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.819870:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.819872:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.819874:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7e200. 00000100:00000001:6.0:1496184948.819875:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.819876:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.819877:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e779800. 02000000:00000001:6.0:1496184948.819878:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.819880:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.819881:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.819884:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.819889:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.819903:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.819904:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.819904:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.819905:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.819907:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.819913:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23942us (24103us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.819918:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57653 00000100:00000040:6.0:1496184948.819920:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.819921:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.819922:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.819925:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.819930:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.819931:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.819933:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.819934:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.819935:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.819937:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.819938:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.819939:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.819939:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.819940:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00002000:00000001:6.0:1496184948.819941:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.819943:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.819944:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5480. 00000020:00000010:6.0:1496184948.819946:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.819947:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.819949:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.819950:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.819952:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.819957:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.819958:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.830233:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.830235:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.830237:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3850 x1568861104105856 msgsize 520 00000100:00100000:7.0:1496184948.830242:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.830244:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.830246:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.830280:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.830283:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.830285:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.830286:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.830287:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.830288:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.830291:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.830292:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105856 00000020:00000001:6.0:1496184948.830294:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.830295:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.830296:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.830297:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.830299:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.830300:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.830302:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.830303:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.830305:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.830307:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5a00. 00000020:00000010:6.0:1496184948.830308:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.830310:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.830311:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.830312:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.830314:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.830317:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c3850 x1568861104105856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.830322:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.830323:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830327:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.830329:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.830330:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.830331:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.830332:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.830332:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.830333:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.830335:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43392 00000100:00000040:6.0:1496184948.830337:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.830338:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676224592 : -131872033327024 : ffff8810264c3850) 00000100:00000040:6.0:1496184948.830340:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3850 x1568861104105856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.830346:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.830346:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.830348:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105856:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.830350:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105856 00000020:00000001:6.0:1496184948.830351:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.830353:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.830356:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.830359:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.830360:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.830361:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e779800. 02000000:00000001:6.0:1496184948.830362:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830363:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.830366:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.830368:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.830369:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.830371:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.830372:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.830374:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.830377:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.830378:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.830379:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.830380:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.830381:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.830382:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.830383:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.830384:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.830386:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.830386:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.830387:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.830389:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.830390:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.830393:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.830394:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.830395:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.830397:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.830398:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.830400:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.830401:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.830402:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.830403:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.830405:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.830406:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.830407:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.830409:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.830410:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.830411:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.830413:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.830414:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.830415:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.830416:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.830417:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.830418:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.830419:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.830421:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.830422:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.830423:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830424:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.830427:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.830429:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.830430:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.830432:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.830433:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.830434:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830435:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.830437:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.830438:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.830439:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.830441:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.830442:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830443:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.830444:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.830445:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.830446:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.830447:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.830448:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.830449:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.830451:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.830452:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.830453:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.830455:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.830456:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.830457:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.830459:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.830459:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.830460:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.830461:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.830462:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.830463:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.830464:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.830466:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105856 00010000:00000001:6.0:1496184948.830467:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.830469:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c3850 x1568861104105856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.830474:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.830475:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.830476:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.830478:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.830480:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.830481:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.830482:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.830484:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.830484:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830485:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.830487:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.830489:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105856, offset 0 00000100:00000001:6.0:1496184948.830500:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.830501:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.830502:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.830504:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.830505:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.830506:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.830507:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.830508:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3850 x1568861104105856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.830514:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105856:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (275us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.830518:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43392 00000100:00000040:6.0:1496184948.830520:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.830521:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.830522:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.830523:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5a00. 00000020:00000010:6.0:1496184948.830525:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.830527:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.830528:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.830529:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.830531:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.830534:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.830535:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.830585:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.830587:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.830588:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e779800. 02000000:00000001:7.0:1496184948.830591:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.830592:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.831041:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.831042:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.831045:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.831050:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.831051:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.831053:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.831083:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.831085:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.831087:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.831088:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.831089:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.831090:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.831092:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.831094:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.831096:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.831096:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.831098:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.831099:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.831100:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.831102:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.831104:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.831105:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.831107:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.831109:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5800. 00000020:00000010:6.0:1496184948.831110:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.831112:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.831113:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.831114:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.831115:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.831116:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.831117:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.831119:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.831120:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.831121:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.831122:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.831123:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.831124:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.831125:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.831126:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.831127:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.831127:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831128:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.831130:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831131:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831132:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.831133:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.831134:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.831135:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.831138:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.831144:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.831145:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.831147:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.831148:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.831149:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.831150:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.831152:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.831152:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.831153:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.831154:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00002000:00000001:6.0:1496184948.831155:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.831156:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.831158:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.831163:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.831165:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.831166:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.831167:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.831167:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.831168:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.831169:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.831172:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57654 00000100:00000040:6.0:1496184948.831173:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.831174:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671286864 : -131872038264752 : ffff88102600e050) 00000100:00000040:6.0:1496184948.831176:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.831182:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.831182:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.831184:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.831186:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.831187:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.831189:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.831190:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831191:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.831191:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.831193:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.831194:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.831195:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.831196:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.831196:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831197:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.831199:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.831200:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.831201:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184948.831202:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.831204:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831205:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.831206:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.831208:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.831212:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.831214:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.831215:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.831217:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.831219:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.831220:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.831221:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.831222:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.831224:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.831225:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.831227:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.831228:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.831229:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.831230:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.831231:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.831232:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.831233:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.831234:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.831236:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.831260:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.831265:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831267:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.831269:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.831271:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.831272:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.831273:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.831274:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.831275:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7de00. 00000100:00000010:6.0:1496184948.831277:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.831278:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.831282:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.831283:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.831310:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe990-0x592def0cfe990 00000100:00000001:6.0:1496184948.831312:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.832813:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.832816:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.843863:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.843868:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.843872:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.854905:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.854967:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000100:00000001:0.0:1496184948.854968:0:57:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:0.0:1496184948.854971:0:57:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:0.0:1496184948.854974:0:57:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8c050 x1568861104105888 msgsize 520 00010000:00000001:6.0:1496184948.854987:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.854989:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00100000:0.0:1496184948.854989:0:57:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:0.0:1496184948.854990:0:57:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00002000:00000001:6.0:1496184948.854993:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00000100:00000001:0.0:1496184948.854993:0:57:0:(events.c:383:request_in_callback()) Process leaving 00002000:00000001:6.0:1496184948.854994:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.854996:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.854998:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00004000:00000001:2.0:1496184948.855029:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00000001:00000001:6.0:1496184948.855031:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.855033:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00004000:00000001:2.0:1496184948.855033:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00002000:00000001:6.0:1496184948.855035:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.855036:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.855036:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:2.0:1496184948.855037:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 00002000:00000001:6.0:1496184948.855038:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.855039:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.855040:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 02000000:00000001:2.0:1496184948.855040:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000020:00000001:6.0:1496184948.855041:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.855041:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000100:00000001:2.0:1496184948.855042:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000020:00000040:6.0:1496184948.855043:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.855044:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000001:2.0:1496184948.855044:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000100:00000010:6.0:1496184948.855046:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7de00. 00000100:00000001:6.0:1496184948.855047:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.855048:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000001:2.0:1496184948.855048:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:6.0:1496184948.855050:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184948.855051:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00100000:2.0:1496184948.855052:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105888 00000020:02000400:6.0:1496184948.855053:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.855055:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:2.0:1496184948.855055:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00010000:00000080:6.0:1496184948.855057:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184948.855057:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:2.0:1496184948.855059:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:2.0:1496184948.855061:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000040:6.0:1496184948.855062:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000001:2.0:1496184948.855063:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00010000:00000001:6.0:1496184948.855064:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.855064:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.855065:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.855065:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.855066:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.855068:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:2.0:1496184948.855070:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184948.855071:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:2.0:1496184948.855073:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88104fbcb600. 00000100:00100000:6.0:1496184948.855074:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23891us (24028us total) trans 0 rc -110/-999 00000020:00000010:2.0:1496184948.855078:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff881045a05a80. 00000100:00100000:6.0:1496184948.855079:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57654 00000020:00000010:2.0:1496184948.855080:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881010ac6600. 00000100:00000040:6.0:1496184948.855081:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.855082:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.855083:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000001:2.0:1496184948.855083:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:2.0:1496184948.855084:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00002000:00100000:6.0:1496184948.855086:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:2.0:1496184948.855088:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00002000:00000001:6.0:1496184948.855091:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.855092:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.855094:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000100:00000001:2.0:1496184948.855094:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000020:00000001:6.0:1496184948.855095:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.855096:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.855098:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.855099:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.855100:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.855100:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00000100:00000200:2.0:1496184948.855100:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8c050 x1568861104105888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00010000:6.0:1496184948.855101:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00002000:00000001:6.0:1496184948.855102:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.855104:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:2.0:1496184948.855105:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000020:00000010:6.0:1496184948.855106:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5800. 00000020:00000010:6.0:1496184948.855107:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000100:00000001:2.0:1496184948.855107:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:6.0:1496184948.855109:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.855110:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:6.0:1496184948.855112:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184948.855112:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.855114:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:2.0:1496184948.855116:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184948.855118:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.855119:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184948.855119:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184948.855120:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00004000:00000001:6.0:1496184948.855121:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:2.0:1496184948.855121:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184948.855122:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:2.0:1496184948.855126:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6101 00000100:00000040:2.0:1496184948.855128:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184948.855129:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736396368 : -131906973155248 : ffff880803b8c050) 00000100:00000040:2.0:1496184948.855132:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8c050 x1568861104105888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184948.855137:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184948.855138:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:2.0:1496184948.855140:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105888:12345-10.37.248.124@o2ib1:8 00000100:00000200:2.0:1496184948.855143:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105888 00000020:00000001:2.0:1496184948.855145:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:2.0:1496184948.855149:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:2.0:1496184948.855164:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:2.0:1496184948.855169:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:2.0:1496184948.855171:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:2.0:1496184948.855173:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88081f7c3400. 02000000:00000001:2.0:1496184948.855175:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855177:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184948.855183:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:2.0:1496184948.855185:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:2.0:1496184948.855188:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:2.0:1496184948.855190:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:2.0:1496184948.855193:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:2.0:1496184948.855196:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:2.0:1496184948.855199:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:2.0:1496184948.855202:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:2.0:1496184948.855205:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:2.0:1496184948.855207:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:2.0:1496184948.855209:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:2.0:1496184948.855210:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:2.0:1496184948.855212:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:2.0:1496184948.855214:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:2.0:1496184948.855216:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:2.0:1496184948.855217:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:2.0:1496184948.855219:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:2.0:1496184948.855220:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:2.0:1496184948.855222:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:2.0:1496184948.855227:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:2.0:1496184948.855228:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:2.0:1496184948.855230:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:2.0:1496184948.855232:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:2.0:1496184948.855234:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:2.0:1496184948.855236:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:2.0:1496184948.855238:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:2.0:1496184948.855239:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:2.0:1496184948.855240:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:2.0:1496184948.855243:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:2.0:1496184948.855244:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:2.0:1496184948.855246:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184948.855247:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:2.0:1496184948.855249:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:2.0:1496184948.855251:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:2.0:1496184948.855253:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184948.855254:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184948.855256:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184948.855258:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184948.855260:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184948.855261:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184948.855262:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184948.855264:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184948.855267:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184948.855268:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855270:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:2.0:1496184948.855273:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:2.0:1496184948.855284:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184948.855286:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:2.0:1496184948.855289:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184948.855290:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184948.855291:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855292:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184948.855294:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184948.855295:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184948.855297:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1496184948.855299:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1496184948.855301:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855303:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:2.0:1496184948.855305:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:2.0:1496184948.855306:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1496184948.855308:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:2.0:1496184948.855309:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:2.0:1496184948.855310:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:2.0:1496184948.855312:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:2.0:1496184948.855314:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184948.855315:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184948.855318:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:2.0:1496184948.855319:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.855322:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:2.0:1496184948.855324:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:2.0:1496184948.855326:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:2.0:1496184948.855327:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1496184948.855328:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:2.0:1496184948.855329:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:2.0:1496184948.855331:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.855332:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:2.0:1496184948.855334:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:2.0:1496184948.855336:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105888 00010000:00000001:2.0:1496184948.855338:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:2.0:1496184948.855341:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8c050 x1568861104105888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:2.0:1496184948.855348:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:2.0:1496184948.855349:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1496184948.855350:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=102 v=1 (1 1 1 1) 00000100:00001000:2.0:1496184948.855353:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:2.0:1496184948.855355:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184948.855357:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184948.855358:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184948.855359:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:2.0:1496184948.855361:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855362:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:2.0:1496184948.855364:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:2.0:1496184948.855368:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105888, offset 0 00000100:00000001:2.0:1496184948.855393:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184948.855394:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184948.855395:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184948.855396:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184948.855434:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:2.0:1496184948.855436:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184948.855437:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:2.0:1496184948.855440:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8c050 x1568861104105888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184948.855447:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105888:12345-10.37.248.124@o2ib1:8 Request procesed in 308us (471us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184948.855452:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6101 00000100:00000040:2.0:1496184948.855454:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:2.0:1496184948.855455:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:2.0:1496184948.855456:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:2.0:1496184948.855459:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff881045a05a80. 00000020:00000010:2.0:1496184948.855462:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881010ac6600. 00000020:00000010:2.0:1496184948.855464:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88104fbcb600. 00000020:00000040:2.0:1496184948.855465:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:2.0:1496184948.855468:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:2.0:1496184948.855470:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:2.0:1496184948.855476:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184948.855477:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.856138:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.856141:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.856143:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.856144:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.856145:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.856146:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.856149:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.856151:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.856153:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.856154:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.856155:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.856156:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.856158:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.856159:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.856162:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.856163:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.856164:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.856167:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5f00. 00000020:00000010:6.0:1496184948.856169:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.856171:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.856172:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.856173:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.856174:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.856175:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.856176:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.856178:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.856179:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.856180:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.856181:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.856183:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.856191:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.856192:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.856193:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.856194:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.856195:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856196:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.856197:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856198:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856199:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.856200:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.856201:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.856202:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.856205:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.856212:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.856213:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.856214:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.856215:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.856216:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.856218:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.856219:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.856220:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.856220:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.856221:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00002000:00000001:6.0:1496184948.856222:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.856223:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.856226:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.856230:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.856232:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.856233:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.856234:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.856235:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.856236:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.856237:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.856240:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57655 00000100:00000040:6.0:1496184948.856241:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.856242:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671287888 : -131872038263728 : ffff88102600e450) 00000100:00000040:6.0:1496184948.856245:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.856250:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.856250:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.856252:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.856254:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.856256:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.856257:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.856258:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856259:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.856260:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.856261:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.856262:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.856264:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.856264:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.856265:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856266:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.856268:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.856269:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.856270:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184948.856272:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.856273:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856274:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.856275:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.856277:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.856281:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.856284:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.856285:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.856287:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.856289:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.856290:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.856291:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.856292:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.856294:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.856295:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.856297:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.856299:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.856299:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.856300:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.856301:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.856303:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.856303:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.856304:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.856307:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.856327:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.856333:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856335:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.856337:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.856338:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.856339:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.856340:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.856342:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.856343:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7de00. 00000100:00000010:6.0:1496184948.856344:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.856346:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.856350:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.856351:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.856377:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe9b0-0x592def0cfe9b0 00000100:00000001:6.0:1496184948.856379:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.857990:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.857992:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.868973:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.868978:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.868979:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184948.879948:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.880009:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.880029:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.880031:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.880035:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.880036:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.880038:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.880039:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.880072:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.880075:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.880077:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.880077:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.880078:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.880079:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.880081:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.880082:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.880083:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.880083:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.880085:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.880086:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.880088:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7de00. 00000100:00000001:6.0:1496184948.880089:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.880090:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.880091:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77f000. 02000000:00000001:6.0:1496184948.880092:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.880094:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.880096:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.880098:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.880104:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.880105:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.880106:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.880107:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.880108:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.880110:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.880116:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23864us (24016us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.880121:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57655 00000100:00000040:6.0:1496184948.880123:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.880124:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.880125:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.880127:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.880133:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.880133:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.880135:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.880136:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.880137:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.880139:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.880140:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.880141:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.880141:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.880142:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00002000:00000001:6.0:1496184948.880143:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.880145:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.880146:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5f00. 00000020:00000010:6.0:1496184948.880148:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.880150:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.880151:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.880153:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.880155:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.880160:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.880161:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.891577:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.891579:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.891582:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c3450 x1568861104105920 msgsize 520 00000100:00100000:7.0:1496184948.891587:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.891588:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.891590:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.891623:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.891626:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.891627:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.891628:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.891629:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.891630:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.891633:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.891635:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105920 00000020:00000001:6.0:1496184948.891636:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.891637:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.891638:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.891639:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.891641:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.891643:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.891645:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.891645:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.891647:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.891649:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5400. 00000020:00000010:6.0:1496184948.891650:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.891652:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.891653:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.891654:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.891656:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.891659:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c3450 x1568861104105920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.891664:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.891665:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891670:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.891672:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.891673:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.891673:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.891674:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.891675:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.891676:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.891678:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43393 00000100:00000040:6.0:1496184948.891680:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.891681:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676223568 : -131872033328048 : ffff8810264c3450) 00000100:00000040:6.0:1496184948.891683:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c3450 x1568861104105920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.891689:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.891689:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.891691:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105920:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.891693:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105920 00000020:00000001:6.0:1496184948.891694:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.891696:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.891699:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.891702:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.891703:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.891704:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77f000. 02000000:00000001:6.0:1496184948.891705:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891707:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.891709:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.891711:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.891713:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.891714:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.891716:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.891718:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.891721:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.891722:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.891723:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.891725:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.891726:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.891726:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.891728:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.891729:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.891731:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.891732:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.891733:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.891734:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.891735:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.891739:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.891739:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.891741:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.891743:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.891743:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.891745:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.891747:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.891748:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.891749:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.891751:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.891752:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.891754:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.891755:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.891756:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.891758:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.891759:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.891760:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.891761:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.891763:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.891764:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.891765:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.891766:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.891768:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.891769:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.891771:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891772:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.891774:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.891777:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.891778:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.891780:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.891781:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.891782:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891784:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.891785:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.891786:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.891788:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.891789:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.891790:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891792:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.891793:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.891794:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.891795:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.891796:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.891797:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.891798:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.891800:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.891801:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.891803:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.891804:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.891806:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.891807:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.891808:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.891809:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.891810:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.891811:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.891812:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.891813:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.891814:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.891815:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105920 00010000:00000001:6.0:1496184948.891817:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.891819:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c3450 x1568861104105920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.891824:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.891825:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.891826:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.891828:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.891830:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.891831:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.891833:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.891834:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.891835:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891836:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.891838:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.891840:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105920, offset 0 00000100:00000001:6.0:1496184948.891850:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.891852:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.891853:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.891854:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.891855:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.891856:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.891857:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.891858:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c3450 x1568861104105920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.891864:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105920:12345-10.37.248.124@o2ib1:8 Request procesed in 174us (281us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.891868:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43393 00000100:00000040:6.0:1496184948.891870:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.891871:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.891872:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.891873:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5400. 00000020:00000010:6.0:1496184948.891875:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.891876:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.891878:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.891879:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.891881:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.891884:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.891885:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.891941:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.891943:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.891944:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77f000. 02000000:00000001:7.0:1496184948.891947:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.891948:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.892369:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.892371:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.892373:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.892378:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.892379:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.892381:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.892412:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.892415:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.892416:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.892417:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.892418:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.892420:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.892422:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.892424:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.892425:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.892426:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.892427:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.892428:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.892430:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.892432:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.892434:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.892434:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.892436:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.892438:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5180. 00000020:00000010:6.0:1496184948.892440:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.892442:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.892444:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.892444:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.892445:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.892446:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.892447:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.892449:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.892450:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.892451:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.892452:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.892453:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.892454:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.892455:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.892456:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.892457:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.892457:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892459:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.892460:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892461:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892462:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.892463:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.892464:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.892465:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.892468:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.892475:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.892475:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.892477:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.892478:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.892479:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.892480:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.892482:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.892482:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.892483:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.892484:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00002000:00000001:6.0:1496184948.892485:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.892486:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.892488:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.892493:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.892495:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.892496:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.892496:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.892497:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.892498:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.892499:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.892501:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57656 00000100:00000040:6.0:1496184948.892503:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.892504:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671284816 : -131872038266800 : ffff88102600d850) 00000100:00000040:6.0:1496184948.892506:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.892512:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.892512:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.892514:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.892516:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.892517:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.892519:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.892520:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892521:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.892522:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.892523:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.892524:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.892525:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.892526:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.892526:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892528:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.892529:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.892530:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.892531:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77fc00. 02000000:00000001:6.0:1496184948.892533:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.892534:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892535:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.892536:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.892538:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.892542:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.892544:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.892545:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.892547:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.892549:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.892550:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.892551:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.892552:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.892554:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.892555:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.892557:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.892558:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.892559:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.892560:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.892561:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.892562:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.892563:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.892564:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.892566:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.892589:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.892595:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892596:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.892599:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.892600:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.892601:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.892602:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.892603:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.892604:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7fe00. 00000100:00000010:6.0:1496184948.892606:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.892607:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.892612:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.892613:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.892639:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe9d0-0x592def0cfe9d0 00000100:00000001:6.0:1496184948.892641:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.894239:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.894242:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00000001:3.0:1496184948.905272:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.905277:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.905278:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000001:3.0:1496184948.916256:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.916323:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.916342:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.916344:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.916348:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.916349:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.916351:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.916353:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.916386:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.916388:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.916390:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.916391:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.916392:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.916393:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.916394:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.916396:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.916396:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.916397:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.916398:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.916400:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.916401:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7fe00. 00000100:00000001:6.0:1496184948.916403:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.916404:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.916405:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77fc00. 02000000:00000001:6.0:1496184948.916406:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.916408:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.916409:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.916412:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.916417:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.916418:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.916419:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.916420:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.916421:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.916423:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.916429:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23916us (24054us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.916434:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57656 00000100:00000040:6.0:1496184948.916435:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.916437:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.916438:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.916440:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.916446:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.916446:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.916448:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.916449:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.916450:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.916452:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.916453:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.916454:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.916454:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.916455:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d850. 00002000:00000001:6.0:1496184948.916456:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.916458:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.916459:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5180. 00000020:00000010:6.0:1496184948.916461:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.916462:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.916464:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.916465:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.916467:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.916472:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.916473:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.927241:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.927243:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184948.927245:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2450 x1568861104105952 msgsize 520 00000100:00100000:7.0:1496184948.927250:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.927251:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.927254:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.927286:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.927289:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.927290:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.927291:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.927292:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.927293:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.927296:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.927298:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105952 00000020:00000001:6.0:1496184948.927299:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.927300:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.927301:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.927303:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.927304:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.927306:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.927308:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.927309:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.927311:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.927312:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5080. 00000020:00000010:6.0:1496184948.927314:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.927316:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.927316:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.927317:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.927319:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.927323:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c2450 x1568861104105952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.927328:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.927328:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927333:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.927335:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.927336:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.927337:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.927338:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.927338:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.927339:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.927341:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43394 00000100:00000040:6.0:1496184948.927343:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.927344:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676219472 : -131872033332144 : ffff8810264c2450) 00000100:00000040:6.0:1496184948.927347:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2450 x1568861104105952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.927352:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.927352:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.927354:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105952:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.927356:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105952 00000020:00000001:6.0:1496184948.927357:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.927359:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.927362:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.927365:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.927366:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.927367:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77fc00. 02000000:00000001:6.0:1496184948.927368:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927369:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.927372:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.927373:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.927375:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.927377:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.927378:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.927380:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.927383:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.927384:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.927385:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.927386:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.927387:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.927388:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.927389:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.927390:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.927392:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.927392:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.927393:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.927395:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.927396:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.927399:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.927400:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.927401:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.927403:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.927404:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.927406:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.927407:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.927408:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.927409:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.927411:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.927412:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.927414:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.927415:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.927416:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.927417:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.927419:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.927420:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.927421:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.927422:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.927423:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.927424:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.927425:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.927426:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.927428:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.927429:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927430:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.927433:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.927435:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.927436:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.927438:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.927439:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.927440:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927441:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.927443:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.927444:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.927445:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.927446:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.927448:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927449:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.927450:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.927451:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.927452:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.927453:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.927454:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.927455:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.927457:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.927458:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.927459:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.927461:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.927462:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.927463:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.927465:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.927465:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.927466:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.927467:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.927468:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.927469:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.927470:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.927472:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105952 00010000:00000001:6.0:1496184948.927473:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.927475:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c2450 x1568861104105952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.927480:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.927481:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.927482:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.927484:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.927486:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.927487:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.927488:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.927490:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.927491:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927492:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.927493:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.927495:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105952, offset 0 00000100:00000001:6.0:1496184948.927506:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.927507:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.927508:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.927509:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.927510:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.927511:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.927512:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.927514:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2450 x1568861104105952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.927520:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105952:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (273us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.927524:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43394 00000100:00000040:6.0:1496184948.927526:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.927527:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.927528:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.927529:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5080. 00000020:00000010:6.0:1496184948.927531:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.927532:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.927533:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.927535:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.927537:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.927541:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.927542:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.928175:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.928177:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.928179:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.928180:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.928181:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.928182:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.928184:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.928186:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.928188:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.928189:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.928190:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.928191:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.928193:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.928194:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.928197:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.928197:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.928199:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.928201:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5b80. 00000020:00000010:6.0:1496184948.928202:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.928204:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.928206:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.928207:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.928207:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.928209:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.928210:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.928211:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.928212:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.928213:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.928214:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.928216:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.928216:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.928218:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.928218:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.928219:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.928220:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928221:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.928222:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928223:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928224:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.928225:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.928226:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.928227:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.928230:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.928237:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.928238:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.928239:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.928240:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.928241:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.928243:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.928244:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.928245:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.928245:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.928246:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00002000:00000001:6.0:1496184948.928247:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.928248:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.928250:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.928255:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.928257:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.928258:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.928259:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.928260:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.928260:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.928261:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.928264:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57657 00000100:00000040:6.0:1496184948.928265:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.928266:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671285840 : -131872038265776 : ffff88102600dc50) 00000100:00000040:6.0:1496184948.928269:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.928274:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.928274:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.928276:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.928278:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.928279:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.928281:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.928282:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928283:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.928284:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.928285:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.928286:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.928287:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.928288:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.928289:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928290:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.928291:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.928292:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.928294:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e778c00. 02000000:00000001:6.0:1496184948.928295:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.928296:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928297:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.928298:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.928300:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.928304:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.928307:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.928308:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.928310:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.928312:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.928313:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.928314:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.928315:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.928317:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.928318:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.928320:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.928321:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.928322:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.928322:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.928323:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.928325:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.928326:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.928327:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.928329:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.928353:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.928359:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928360:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.928363:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.928364:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.928365:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.928366:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.928367:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.928369:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c200. 00000100:00000010:6.0:1496184948.928370:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.928371:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.928376:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.928377:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.928404:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfe9f0-0x592def0cfe9f0 00000100:00000001:6.0:1496184948.928406:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.930029:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.930032:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.941082:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.941088:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.941088:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.952085:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.952149:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.952168:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.952170:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.952174:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.952175:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.952177:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.952179:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.952212:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.952214:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.952216:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.952217:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.952218:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.952219:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.952220:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.952221:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.952222:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.952223:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.952224:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.952225:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.952227:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c200. 00000100:00000001:6.0:1496184948.952229:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.952230:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.952231:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e778c00. 02000000:00000001:6.0:1496184948.952232:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.952234:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.952236:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.952238:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.952243:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.952245:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.952245:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.952246:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.952247:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.952249:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.952255:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23980us (24127us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.952260:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57657 00000100:00000040:6.0:1496184948.952262:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.952263:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.952264:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.952266:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600dc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.952272:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.952272:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600dc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.952274:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.952275:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.952276:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.952278:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.952279:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.952280:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.952281:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.952281:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600dc50. 00002000:00000001:6.0:1496184948.952282:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.952284:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.952285:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5b80. 00000020:00000010:6.0:1496184948.952287:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.952288:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.952290:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.952291:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.952293:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.952299:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.952300:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.963004:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.963006:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.963008:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.963009:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.963010:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.963011:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.963013:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.963015:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104105984 00000020:00000001:6.0:1496184948.963017:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.963018:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.963019:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.963020:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.963021:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.963023:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.963025:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.963026:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.963027:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.963030:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5a80. 00000020:00000010:6.0:1496184948.963031:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.963033:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.963034:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.963035:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.963037:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.963040:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c2c50 x1568861104105984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.963045:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.963046:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963050:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.963053:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.963053:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.963054:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.963055:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.963056:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.963056:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.963059:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43395 00000100:00000040:6.0:1496184948.963060:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.963062:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676221520 : -131872033330096 : ffff8810264c2c50) 00000100:00000040:6.0:1496184948.963064:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2c50 x1568861104105984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.963069:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.963070:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.963072:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105984:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.963074:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104105984 00000020:00000001:6.0:1496184948.963075:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.963077:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.963080:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.963083:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.963084:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.963085:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e778c00. 02000000:00000001:6.0:1496184948.963086:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963087:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.963090:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.963091:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.963093:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.963094:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.963096:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.963098:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.963100:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.963102:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.963103:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.963104:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.963105:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.963105:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.963107:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.963108:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.963109:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.963110:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.963111:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.963112:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.963113:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.963117:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.963117:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.963119:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.963121:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.963121:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.963123:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.963125:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.963125:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.963126:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.963128:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.963130:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.963131:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.963132:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.963133:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.963135:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.963136:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.963137:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.963138:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.963139:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.963141:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.963142:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.963143:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.963144:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.963145:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.963147:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963148:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.963150:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.963152:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.963154:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.963156:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.963157:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.963158:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963159:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.963160:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.963161:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.963163:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.963164:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.963165:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963166:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.963167:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.963168:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.963169:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.963171:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.963171:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.963173:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.963174:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.963175:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.963176:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.963178:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963179:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.963180:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.963182:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.963183:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.963184:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.963184:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.963185:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963186:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.963188:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.963189:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104105984 00010000:00000001:6.0:1496184948.963190:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.963192:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c2c50 x1568861104105984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.963197:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.963198:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.963199:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.963201:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.963203:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.963204:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.963206:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.963207:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.963208:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963209:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.963210:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.963212:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104105984, offset 0 00000100:00000001:6.0:1496184948.963223:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963224:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.963226:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.963227:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.963228:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.963228:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963229:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.963231:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2c50 x1568861104105984/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.963240:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104105984:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (277us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.963244:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43395 00000100:00000040:6.0:1496184948.963246:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.963247:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.963248:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.963249:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5a80. 00000020:00000010:6.0:1496184948.963251:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.963252:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.963254:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.963255:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.963257:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.963260:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.963261:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.963846:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.963849:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.963850:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.963851:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.963852:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.963853:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.963856:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.963858:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.963859:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.963860:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.963861:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.963862:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.963864:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.963865:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.963868:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.963868:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.963870:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.963872:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d5e5f80. 00000020:00000010:6.0:1496184948.963874:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.963876:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.963878:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.963878:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.963879:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.963880:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.963881:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.963883:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.963884:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.963885:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.963886:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.963887:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.963888:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.963906:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.963907:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.963908:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.963909:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963910:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.963911:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963912:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963914:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.963915:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.963916:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.963917:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.963920:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.963926:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.963927:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.963929:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.963930:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.963931:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.963932:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.963934:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.963934:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.963935:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.963935:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00002000:00000001:6.0:1496184948.963937:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963938:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963940:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963944:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.963946:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.963947:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.963948:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.963948:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.963949:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.963950:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.963953:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57658 00000100:00000040:6.0:1496184948.963954:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.963955:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671293008 : -131872038258608 : ffff88102600f850) 00000100:00000040:6.0:1496184948.963958:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.963963:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.963963:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.963965:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.963967:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.963968:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.963970:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.963971:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963972:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.963973:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.963974:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.963975:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.963976:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.963977:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.963978:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963979:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.963981:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.963981:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.963983:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77a400. 02000000:00000001:6.0:1496184948.963984:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.963985:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.963987:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.963987:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.963989:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.963994:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.963996:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.963997:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.963999:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.964001:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.964002:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.964003:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.964004:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.964006:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.964007:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.964009:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.964010:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.964011:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.964012:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.964013:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.964014:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.964015:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.964016:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.964019:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.964040:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.964046:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.964047:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.964050:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.964051:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.964052:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.964053:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.964054:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.964055:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7c200. 00000100:00000010:6.0:1496184948.964057:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.964058:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.964063:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.964064:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.964090:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfea10-0x592def0cfea10 00000100:00000001:6.0:1496184948.964092:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184948.965714:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.965717:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.976754:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184948.976759:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184948.976760:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7c200 00000100:00000001:3.0:1496184948.987746:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184948.987812:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184948.987831:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.987833:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184948.987837:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184948.987838:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.987840:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.987841:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184948.987874:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184948.987876:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184948.987878:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184948.987879:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184948.987880:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184948.987881:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.987882:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184948.987884:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184948.987884:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184948.987885:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184948.987886:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184948.987888:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184948.987900:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7c200. 00000100:00000001:6.0:1496184948.987901:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184948.987902:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184948.987903:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77a400. 02000000:00000001:6.0:1496184948.987904:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184948.987906:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184948.987908:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184948.987910:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184948.987916:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184948.987917:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184948.987918:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.987919:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.987920:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.987922:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184948.987928:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23963us (24117us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184948.987933:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57658 00000100:00000040:6.0:1496184948.987934:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.987936:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184948.987937:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.987939:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600f850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184948.987945:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.987945:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600f850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.987947:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.987948:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.987949:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.987951:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.987952:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.987953:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.987953:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.987954:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600f850. 00002000:00000001:6.0:1496184948.987955:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.987956:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.987958:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d5e5f80. 00000020:00000010:6.0:1496184948.987959:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.987961:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.987963:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.987964:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.987966:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184948.987971:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.987973:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.998553:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.998555:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.998557:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.998558:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.998559:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.998560:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.998562:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.998564:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106016 00000020:00000001:6.0:1496184948.998566:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.998567:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.998568:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.998569:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.998570:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.998572:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.998574:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.998575:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.998576:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.998579:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88007d684580. 00000020:00000010:6.0:1496184948.998580:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184948.998582:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.998583:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184948.998584:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184948.998586:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184948.998589:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c2850 x1568861104106016/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.998594:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184948.998595:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998599:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.998601:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.998602:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.998603:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.998604:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.998605:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.998605:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.998608:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43396 00000100:00000040:6.0:1496184948.998609:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.998610:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676220496 : -131872033331120 : ffff8810264c2850) 00000100:00000040:6.0:1496184948.998613:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c2850 x1568861104106016/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184968 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.998618:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.998619:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.998620:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106016:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184948.998622:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106016 00000020:00000001:6.0:1496184948.998624:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184948.998626:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184948.998629:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184948.998631:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.998632:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.998634:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77a400. 02000000:00000001:6.0:1496184948.998635:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998636:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.998638:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.998640:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184948.998642:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184948.998643:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184948.998645:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184948.998646:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184948 last 1496184947 00010000:00000001:6.0:1496184948.998649:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184948.998650:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184948.998651:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184948.998652:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184948.998653:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184948.998654:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184948.998655:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184948.998657:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184948.998658:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184948.998659:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184948.998660:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184948.998661:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184948.998662:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184948.998665:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.998666:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.998667:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.998669:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.998670:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.998672:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.998674:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184948.998674:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184948.998675:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184948.998677:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184948.998679:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184948.998680:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.998681:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184948.998682:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184948.998684:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184948.998685:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.998686:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.998687:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184948.998688:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184948.998690:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.998691:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184948.998692:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.998693:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.998694:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.998696:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998697:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184948.998699:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184948.998702:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184948.998703:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184948.998705:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.998706:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.998707:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998708:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.998709:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.998711:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.998712:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184948.998713:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184948.998714:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998716:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184948.998717:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184948.998717:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184948.998718:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184948.998720:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184948.998720:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184948.998722:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184948.998723:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184948.998725:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184948.998726:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184948.998727:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.998728:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.998730:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184948.998731:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.998732:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.998733:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.998734:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.998735:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.998736:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184948.998737:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184948.998738:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106016 00010000:00000001:6.0:1496184948.998740:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184948.998742:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c2850 x1568861104106016/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184948.998747:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184948.998748:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184948.998749:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=93 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184948.998751:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184948.998753:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184948.998754:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.998755:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184948.998756:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184948.998757:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998758:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184948.998760:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184948.998763:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106016, offset 0 00000100:00000001:6.0:1496184948.998774:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.998775:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184948.998777:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184948.998778:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184948.998779:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184948.998779:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.998780:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184948.998782:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c2850 x1568861104106016/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184968 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184948.998788:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106016:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (274us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184948.998792:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43396 00000100:00000040:6.0:1496184948.998794:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184948.998795:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184948.998796:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184948.998797:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88007d684580. 00000020:00000010:6.0:1496184948.998799:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184948.998801:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184948.998802:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184948.998804:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184948.998805:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184948.998809:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.998810:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184948.998824:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184948.998826:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184948.998828:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e77a400. 02000000:00000001:7.0:1496184948.998831:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184948.998832:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184948.999339:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184948.999340:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184948.999343:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ec50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184948.999348:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184948.999349:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184948.999351:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:6.0:1496184948.999383:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.999385:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.999387:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184948.999388:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184948.999389:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184948.999390:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184948.999393:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184948.999394:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184948.999396:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184948.999397:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184948.999398:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184948.999399:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184948.999401:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184948.999402:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184948.999404:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.999405:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184948.999407:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184948.999409:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085c69e680. 00000020:00000010:6.0:1496184948.999410:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184948.999412:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184948.999414:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184948.999415:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184948.999416:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184948.999417:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.999418:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.999419:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.999420:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184948.999422:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184948.999423:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.999424:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184948.999425:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.999426:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.999427:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.999428:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184948.999428:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999429:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184948.999430:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999431:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999432:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.999434:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184948.999435:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.999436:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184948.999439:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184948.999445:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184948.999446:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184948.999448:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184948.999448:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184948.999449:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.999451:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184948.999452:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184948.999453:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184948.999453:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184948.999454:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00002000:00000001:6.0:1496184948.999455:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.999456:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.999459:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.999463:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184948.999465:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184948.999466:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184948.999467:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184948.999468:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184948.999469:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184948.999470:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184948.999472:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57659 00000100:00000040:6.0:1496184948.999473:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184948.999475:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671289936 : -131872038261680 : ffff88102600ec50) 00000100:00000040:6.0:1496184948.999477:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/0 e 0 to 0 dl 1496184991 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184948.999482:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184948.999483:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184948.999484:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184948.999486:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184948.999487:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184948.999489:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184948.999490:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999491:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184948.999492:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184948.999493:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184948.999494:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184948.999495:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184948.999496:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184948.999497:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999498:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184948.999500:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184948.999500:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184948.999502:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77bc00. 02000000:00000001:6.0:1496184948.999503:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.999504:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999505:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184948.999506:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184948.999508:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184948.999512:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184948.999515:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184948.999516:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184948.999518:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184948.999520:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184948.999521:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184948.999522:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184948.999523:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184948.999525:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184948.999526:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184948.999528:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184948.999529:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184948.999530:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184948.999530:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184948.999531:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184948.999533:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184948.999534:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184948.999535:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184948.999537:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184948.999559:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184948.999564:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999566:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184948.999568:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.999570:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184948.999571:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184948.999572:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184948.999573:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184948.999574:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7fe00. 00000100:00000010:6.0:1496184948.999576:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184948.999577:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184948.999581:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184948.999582:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184948.999608:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfea30-0x592def0cfea30 00000100:00000001:6.0:1496184948.999611:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.001208:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.001222:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7fe00 00000100:00000001:3.0:1496184949.012240:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.012245:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.012246:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7fe00 00000100:00000001:3.0:1496184949.023230:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184949.023291:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184949.023310:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184949.023313:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:6.0:1496184949.023316:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184949.023317:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184949.023320:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184949.023321:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:6.0:1496184949.023354:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184949.023357:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184949.023359:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184949.023360:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:6.0:1496184949.023360:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184949.023362:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184949.023363:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184949.023364:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184949.023365:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184949.023365:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184949.023367:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:6.0:1496184949.023368:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184949.023370:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7fe00. 00000100:00000001:6.0:1496184949.023371:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184949.023372:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.023373:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77bc00. 02000000:00000001:6.0:1496184949.023374:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184949.023376:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184949.023378:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:6.0:1496184949.023380:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:6.0:1496184949.023385:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:6.0:1496184949.023387:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184949.023387:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184949.023388:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.023389:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184949.023391:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:6.0:1496184949.023401:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23918us (24056us total) trans 0 rc -110/-999 00000100:00100000:6.0:1496184949.023406:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57659 00000100:00000040:6.0:1496184949.023408:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184949.023409:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184949.023410:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184949.023413:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ec50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:736/0 lens 608/448 e 0 to 0 dl 1496184991 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:6.0:1496184949.023418:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184949.023419:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ec50 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184949.023420:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184949.023421:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184949.023422:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.023424:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.023425:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184949.023426:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184949.023427:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184949.023427:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ec50. 00002000:00000001:6.0:1496184949.023428:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.023430:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184949.023431:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085c69e680. 00000020:00000010:6.0:1496184949.023433:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184949.023434:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184949.023436:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184949.023437:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184949.023439:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:6.0:1496184949.023444:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184949.023445:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184949.033824:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184949.033827:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184949.033828:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184949.033830:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184949.033831:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184949.033832:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184949.033834:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184949.033836:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106048 00000020:00000001:6.0:1496184949.033838:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184949.033838:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184949.033840:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184949.033841:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184949.033842:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184949.033844:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184949.033846:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184949.033847:0:13731:0:(service.c:1128:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184949.033849:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184949.033851:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085c69e380. 00000020:00000010:6.0:1496184949.033852:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000001:6.0:1496184949.033854:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184949.033855:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:6.0:1496184949.033856:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184949.033858:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:6.0:1496184949.033861:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c1c50 x1568861104106048/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184949.033866:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:6.0:1496184949.033866:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.033871:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184949.033873:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184949.033874:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184949.033875:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184949.033875:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184949.033876:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184949.033877:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184949.033879:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43397 00000100:00000040:6.0:1496184949.033881:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184949.033882:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676217424 : -131872033334192 : ffff8810264c1c50) 00000100:00000040:6.0:1496184949.033884:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c1c50 x1568861104106048/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:6.0:1496184949.033902:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184949.033903:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184949.033905:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106048:12345-10.37.248.124@o2ib1:8 00000100:00000200:6.0:1496184949.033907:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106048 00000020:00000001:6.0:1496184949.033908:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:6.0:1496184949.033910:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:6.0:1496184949.033913:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:6.0:1496184949.033916:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184949.033917:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184949.033918:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e77bc00. 02000000:00000001:6.0:1496184949.033919:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.033921:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184949.033923:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184949.033925:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:6.0:1496184949.033926:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:6.0:1496184949.033928:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:6.0:1496184949.033929:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:6.0:1496184949.033931:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:6.0:1496184949.033934:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:6.0:1496184949.033935:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:6.0:1496184949.033936:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:6.0:1496184949.033937:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:6.0:1496184949.033938:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:6.0:1496184949.033939:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:6.0:1496184949.033940:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:6.0:1496184949.033941:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:6.0:1496184949.033943:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:6.0:1496184949.033944:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:6.0:1496184949.033945:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:6.0:1496184949.033946:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:6.0:1496184949.033947:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:6.0:1496184949.033950:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184949.033951:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184949.033952:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184949.033954:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184949.033955:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184949.033957:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184949.033959:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:6.0:1496184949.033959:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:6.0:1496184949.033960:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:6.0:1496184949.033962:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:6.0:1496184949.033964:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:6.0:1496184949.033965:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.033966:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:6.0:1496184949.033967:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:6.0:1496184949.033969:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:6.0:1496184949.033970:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.033971:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184949.033972:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:6.0:1496184949.033973:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:6.0:1496184949.033975:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184949.033976:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:6.0:1496184949.033977:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184949.033978:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184949.033979:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184949.033981:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.033982:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:6.0:1496184949.033984:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:6.0:1496184949.033987:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:6.0:1496184949.033988:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:6.0:1496184949.033990:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184949.033991:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184949.033992:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.033993:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184949.033994:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184949.033996:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184949.033997:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:6.0:1496184949.033998:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:6.0:1496184949.033999:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034001:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:6.0:1496184949.034002:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:6.0:1496184949.034002:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:6.0:1496184949.034003:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:6.0:1496184949.034005:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:6.0:1496184949.034005:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:6.0:1496184949.034007:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:6.0:1496184949.034008:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:6.0:1496184949.034010:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:6.0:1496184949.034011:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:6.0:1496184949.034012:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034014:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184949.034015:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:6.0:1496184949.034016:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184949.034017:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184949.034018:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184949.034019:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184949.034020:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034021:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:6.0:1496184949.034022:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:6.0:1496184949.034023:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106048 00010000:00000001:6.0:1496184949.034025:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:6.0:1496184949.034027:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c1c50 x1568861104106048/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:6.0:1496184949.034032:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:6.0:1496184949.034033:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:6.0:1496184949.034034:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:6.0:1496184949.034036:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:6.0:1496184949.034038:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:6.0:1496184949.034039:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184949.034040:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:6.0:1496184949.034042:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:6.0:1496184949.034042:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034043:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:6.0:1496184949.034045:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:6.0:1496184949.034047:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106048, offset 0 00000100:00000001:6.0:1496184949.034058:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034059:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:6.0:1496184949.034060:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:6.0:1496184949.034061:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184949.034062:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184949.034063:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034064:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:6.0:1496184949.034066:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c1c50 x1568861104106048/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:6.0:1496184949.034071:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106048:12345-10.37.248.124@o2ib1:8 Request procesed in 181us (288us total) trans 0 rc 0/0 00000100:00100000:6.0:1496184949.034076:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43397 00000100:00000040:6.0:1496184949.034077:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:6.0:1496184949.034079:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:6.0:1496184949.034079:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184949.034081:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085c69e380. 00000020:00000010:6.0:1496184949.034083:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184949.034084:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184949.034085:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:6.0:1496184949.034087:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:6.0:1496184949.034089:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:6.0:1496184949.034092:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184949.034093:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184949.034597:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184949.034600:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184949.034601:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:6.0:1496184949.034602:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184949.034603:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:6.0:1496184949.034604:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184949.034607:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:6.0:1496184949.034609:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:6.0:1496184949.034610:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:6.0:1496184949.034611:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:6.0:1496184949.034612:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:6.0:1496184949.034613:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:6.0:1496184949.034615:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:6.0:1496184949.034616:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:6.0:1496184949.034618:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184949.034619:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:6.0:1496184949.034621:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000010:6.0:1496184949.034623:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085c69e500. 00000020:00000010:6.0:1496184949.034624:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881038be9cc0. 00000100:00000040:6.0:1496184949.034627:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:6.0:1496184949.034629:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:6.0:1496184949.034629:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184949.034630:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:6.0:1496184949.034631:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184949.034632:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184949.034634:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184949.034635:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:6.0:1496184949.034636:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:6.0:1496184949.034637:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184949.034638:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:6.0:1496184949.034639:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184949.034640:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184949.034641:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184949.034642:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:6.0:1496184949.034642:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034643:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:6.0:1496184949.034645:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034646:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034647:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.034648:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:6.0:1496184949.034649:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.034650:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:6.0:1496184949.034653:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:6.0:1496184949.034659:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184949.034660:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00010000:00000001:6.0:1496184949.034662:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:6.0:1496184949.034663:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:6.0:1496184949.034664:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.034665:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.034666:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184949.034667:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184949.034668:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184949.034668:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00002000:00000001:6.0:1496184949.034669:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034671:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034673:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034677:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184949.034679:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184949.034680:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:6.0:1496184949.034681:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:6.0:1496184949.034682:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:6.0:1496184949.034683:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:6.0:1496184949.034684:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:6.0:1496184949.034686:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57660 00000100:00000040:6.0:1496184949.034688:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:6.0:1496184949.034689:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671723088 : -131872037828528 : ffff881026078850) 00000100:00000040:6.0:1496184949.034691:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:6.0:1496184949.034696:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:6.0:1496184949.034697:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184949.034699:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:6.0:1496184949.034701:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:6.0:1496184949.034702:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:6.0:1496184949.034703:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:6.0:1496184949.034704:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034705:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:6.0:1496184949.034706:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:6.0:1496184949.034707:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:6.0:1496184949.034708:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184949.034709:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:6.0:1496184949.034710:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:6.0:1496184949.034711:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034712:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:6.0:1496184949.034714:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:6.0:1496184949.034714:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:6.0:1496184949.034716:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e77e400. 02000000:00000001:6.0:1496184949.034717:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034718:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034720:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:6.0:1496184949.034720:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:6.0:1496184949.034722:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:6.0:1496184949.034726:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:6.0:1496184949.034729:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:6.0:1496184949.034730:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184949.034732:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184949.034733:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:6.0:1496184949.034734:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:6.0:1496184949.034735:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:6.0:1496184949.034737:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:6.0:1496184949.034738:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:6.0:1496184949.034739:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:6.0:1496184949.034741:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:6.0:1496184949.034743:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:6.0:1496184949.034743:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:6.0:1496184949.034744:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:6.0:1496184949.034745:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:6.0:1496184949.034747:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:6.0:1496184949.034747:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:6.0:1496184949.034748:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:6.0:1496184949.034751:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:6.0:1496184949.034772:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:6.0:1496184949.034777:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034779:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:6.0:1496184949.034781:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.034783:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:6.0:1496184949.034784:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.034785:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.034786:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:6.0:1496184949.034787:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8808f4d7de00. 00000100:00000010:6.0:1496184949.034788:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000020:00000040:6.0:1496184949.034790:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:6.0:1496184949.034794:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:6.0:1496184949.034795:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:6.0:1496184949.034822:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfea50-0x592def0cfea50 00000100:00000001:6.0:1496184949.034824:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.036393:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.036407:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184949.047420:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.047425:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.047426:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8808f4d7de00 00000100:00000001:3.0:1496184949.058455:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:6.0:1496184949.058527:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:6.0:1496184949.058534:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1496184949.058534:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000020:00000001:6.0:1496184949.058536:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000200:1.0:1496184949.058536:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184949.058539:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8cc50 x1568861104106080 msgsize 520 00002000:00000001:6.0:1496184949.058540:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:6.0:1496184949.058541:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:6.0:1496184949.058543:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:6.0:1496184949.058545:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184949.058546:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.058547:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.058550:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00000001:00000001:6.0:1496184949.058578:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:6.0:1496184949.058580:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:6.0:1496184949.058582:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:6.0:1496184949.058583:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00004000:00000001:2.0:1496184949.058583:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00002000:00000001:6.0:1496184949.058584:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:6.0:1496184949.058585:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1496184949.058586:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00004000:00000001:2.0:1496184949.058587:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000020:00000001:6.0:1496184949.058588:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:6.0:1496184949.058588:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:6.0:1496184949.058589:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:6.0:1496184949.058590:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:2.0:1496184949.058591:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 00000100:00000010:6.0:1496184949.058592:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100e7a3000. 00000100:00000010:6.0:1496184949.058593:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8808f4d7de00. 02000000:00000001:2.0:1496184949.058594:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:6.0:1496184949.058595:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:6.0:1496184949.058596:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 00000100:00000001:2.0:1496184949.058596:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 02000000:00000010:6.0:1496184949.058597:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e77e400. 00000100:00000001:2.0:1496184949.058597:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:6.0:1496184949.058598:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:6.0:1496184949.058600:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:6.0:1496184949.058601:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 02000000:00000001:2.0:1496184949.058602:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00010000:00000080:6.0:1496184949.058604:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:2.0:1496184949.058606:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106080 00010000:00000040:6.0:1496184949.058609:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000001:2.0:1496184949.058609:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:2.0:1496184949.058610:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00010000:00000001:6.0:1496184949.058611:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:6.0:1496184949.058611:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:6.0:1496184949.058612:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:6.0:1496184949.058613:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000020:00000001:2.0:1496184949.058613:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000040:6.0:1496184949.058615:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000040:2.0:1496184949.058615:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:00000001:2.0:1496184949.058617:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:2.0:1496184949.058619:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00100000:6.0:1496184949.058621:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23924us (24062us total) trans 0 rc -110/-999 00000100:00000001:2.0:1496184949.058624:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.058625:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:6.0:1496184949.058627:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57660 00000100:00000040:6.0:1496184949.058628:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000010:2.0:1496184949.058628:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100e966e00. 00000100:00000001:6.0:1496184949.058629:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:6.0:1496184949.058630:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000020:00000010:2.0:1496184949.058631:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff881045a05f00. 00002000:00100000:6.0:1496184949.058633:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026078850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000010:2.0:1496184949.058637:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881010ac6600. 00002000:00000001:6.0:1496184949.058639:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:6.0:1496184949.058639:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026078850 with x1568861102817424 ext(0->1048575) 00000100:00000001:2.0:1496184949.058640:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00010000:00000001:6.0:1496184949.058641:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000100:00000001:2.0:1496184949.058641:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:6.0:1496184949.058642:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:2.0:1496184949.058642:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:6.0:1496184949.058643:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.058645:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:6.0:1496184949.058646:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:6.0:1496184949.058647:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:6.0:1496184949.058648:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:6.0:1496184949.058648:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026078850. 00000100:00000001:2.0:1496184949.058648:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00002000:00000001:6.0:1496184949.058650:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.058651:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:6.0:1496184949.058653:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085c69e500. 00000100:00000200:2.0:1496184949.058653:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8cc50 x1568861104106080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000010:6.0:1496184949.058654:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881038be9cc0. 00000020:00000010:6.0:1496184949.058656:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8808f4d7e000. 00000020:00000040:6.0:1496184949.058658:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:6.0:1496184949.058659:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.058659:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:2.0:1496184949.058660:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:6.0:1496184949.058662:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00000001:2.0:1496184949.058666:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:6.0:1496184949.058667:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:6.0:1496184949.058668:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:2.0:1496184949.058670:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.058671:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:2.0:1496184949.058672:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184949.058673:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184949.058675:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184949.058676:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:2.0:1496184949.058679:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6102 00000100:00000040:2.0:1496184949.058681:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.058683:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736399440 : -131906973152176 : ffff880803b8cc50) 00000100:00000040:2.0:1496184949.058686:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8cc50 x1568861104106080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.058691:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.058692:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:2.0:1496184949.058694:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106080:12345-10.37.248.124@o2ib1:8 00000100:00000200:2.0:1496184949.058696:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106080 00000020:00000001:2.0:1496184949.058698:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:2.0:1496184949.058703:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:2.0:1496184949.058717:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:2.0:1496184949.058723:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:2.0:1496184949.058724:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:2.0:1496184949.058727:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88081f7c5000. 02000000:00000001:2.0:1496184949.058729:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058731:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.058737:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:2.0:1496184949.058740:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:2.0:1496184949.058743:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:2.0:1496184949.058745:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:2.0:1496184949.058747:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:2.0:1496184949.058750:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:2.0:1496184949.058754:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:2.0:1496184949.058757:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:2.0:1496184949.058759:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:2.0:1496184949.058761:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:2.0:1496184949.058763:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:2.0:1496184949.058764:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:2.0:1496184949.058766:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:2.0:1496184949.058768:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:2.0:1496184949.058770:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:2.0:1496184949.058772:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:2.0:1496184949.058773:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:2.0:1496184949.058775:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:2.0:1496184949.058776:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:2.0:1496184949.058781:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:2.0:1496184949.058783:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:2.0:1496184949.058784:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:2.0:1496184949.058787:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:2.0:1496184949.058788:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:2.0:1496184949.058791:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:2.0:1496184949.058793:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:2.0:1496184949.058794:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:2.0:1496184949.058795:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:2.0:1496184949.058797:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:2.0:1496184949.058799:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:2.0:1496184949.058801:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.058803:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:2.0:1496184949.058804:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:2.0:1496184949.058806:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:2.0:1496184949.058808:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.058809:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.058810:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.058813:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184949.058815:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.058816:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184949.058817:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.058819:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.058822:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.058823:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058825:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:2.0:1496184949.058828:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:2.0:1496184949.058831:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184949.058833:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:2.0:1496184949.058835:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.058837:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.058838:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058839:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.058840:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.058842:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.058844:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1496184949.058855:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1496184949.058856:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058858:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:2.0:1496184949.058860:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:2.0:1496184949.058861:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1496184949.058863:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:2.0:1496184949.058865:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:2.0:1496184949.058866:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:2.0:1496184949.058867:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:2.0:1496184949.058870:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.058871:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.058873:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:2.0:1496184949.058874:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.058877:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:2.0:1496184949.058879:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:2.0:1496184949.058881:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:2.0:1496184949.058882:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1496184949.058883:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:2.0:1496184949.058885:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:2.0:1496184949.058886:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.058888:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:2.0:1496184949.058898:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:2.0:1496184949.058900:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106080 00010000:00000001:2.0:1496184949.058903:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:2.0:1496184949.058905:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8cc50 x1568861104106080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:2.0:1496184949.058912:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:2.0:1496184949.058913:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1496184949.058915:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:2.0:1496184949.058918:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:2.0:1496184949.058920:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.058921:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.058922:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.058924:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:2.0:1496184949.058925:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058927:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:2.0:1496184949.058929:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:2.0:1496184949.058934:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106080, offset 0 00000100:00000001:2.0:1496184949.058994:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.058995:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.058996:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.058998:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.059000:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:2.0:1496184949.059001:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.059003:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:2.0:1496184949.059005:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8cc50 x1568861104106080/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184949.059011:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106080:12345-10.37.248.124@o2ib1:8 Request procesed in 318us (470us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184949.059016:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6102 00000100:00000040:2.0:1496184949.059018:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:2.0:1496184949.059020:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:2.0:1496184949.059021:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:2.0:1496184949.059024:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff881045a05f00. 00000020:00000010:2.0:1496184949.059026:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881010ac6600. 00000020:00000010:2.0:1496184949.059028:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100e966e00. 00000020:00000040:2.0:1496184949.059030:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:2.0:1496184949.059032:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:2.0:1496184949.059035:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:2.0:1496184949.059041:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.059042:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.059291:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.059293:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.059295:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88081f7c5000. 02000000:00000001:7.0:1496184949.059298:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.059299:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.059712:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.059714:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.059716:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.059721:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.059723:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.059726:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:4.0:1496184949.059757:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184949.059759:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184949.059761:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:4.0:1496184949.059762:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:4.0:1496184949.059764:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:4.0:1496184949.059765:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:4.0:1496184949.059767:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:4.0:1496184949.059769:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:4.0:1496184949.059771:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:4.0:1496184949.059772:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:4.0:1496184949.059773:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:4.0:1496184949.059775:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:4.0:1496184949.059776:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:4.0:1496184949.059778:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:4.0:1496184949.059780:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184949.059781:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:4.0:1496184949.059783:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000010:4.0:1496184949.059785:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed3ff00. 00000020:00000010:4.0:1496184949.059787:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88103ee07a80. 00000100:00000040:4.0:1496184949.059789:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:4.0:1496184949.059791:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:4.0:1496184949.059791:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:4.0:1496184949.059792:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:4.0:1496184949.059794:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184949.059795:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184949.059796:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184949.059797:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:4.0:1496184949.059799:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:4.0:1496184949.059800:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184949.059801:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:4.0:1496184949.059804:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184949.059805:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184949.059806:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184949.059807:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:4.0:1496184949.059807:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059808:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:4.0:1496184949.059810:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059811:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059812:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184949.059813:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:4.0:1496184949.059814:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184949.059815:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184949.059818:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:4.0:1496184949.059824:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184949.059825:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184949.059827:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184949.059828:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184949.059829:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184949.059830:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184949.059832:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184949.059832:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184949.059833:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184949.059834:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00002000:00000001:4.0:1496184949.059835:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.059836:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.059839:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.059843:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:4.0:1496184949.059845:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184949.059846:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:4.0:1496184949.059847:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:4.0:1496184949.059848:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:4.0:1496184949.059849:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:4.0:1496184949.059850:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:4.0:1496184949.059853:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57661 00000100:00000040:4.0:1496184949.059855:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:4.0:1496184949.059856:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671726160 : -131872037825456 : ffff881026079450) 00000100:00000040:4.0:1496184949.059858:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:4.0:1496184949.059864:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:4.0:1496184949.059864:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:4.0:1496184949.059866:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:4.0:1496184949.059868:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:4.0:1496184949.059869:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:4.0:1496184949.059871:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:4.0:1496184949.059872:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059873:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:4.0:1496184949.059874:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:4.0:1496184949.059875:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:4.0:1496184949.059876:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:4.0:1496184949.059877:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:4.0:1496184949.059878:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:4.0:1496184949.059879:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059880:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:4.0:1496184949.059882:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:4.0:1496184949.059883:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:4.0:1496184949.059884:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184949.059885:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.059887:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059888:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:4.0:1496184949.059889:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:4.0:1496184949.059908:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:4.0:1496184949.059913:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:4.0:1496184949.059915:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:4.0:1496184949.059916:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184949.059918:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184949.059921:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:4.0:1496184949.059922:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:4.0:1496184949.059923:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:4.0:1496184949.059925:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:4.0:1496184949.059927:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:4.0:1496184949.059928:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:4.0:1496184949.059930:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:4.0:1496184949.059931:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:4.0:1496184949.059932:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:4.0:1496184949.059933:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:4.0:1496184949.059934:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:4.0:1496184949.059935:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:4.0:1496184949.059936:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:4.0:1496184949.059937:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:4.0:1496184949.059940:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:4.0:1496184949.059965:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:4.0:1496184949.059971:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059972:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:4.0:1496184949.059975:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184949.059976:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:4.0:1496184949.059977:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.059978:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.059979:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:4.0:1496184949.059980:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff8810105bae00. 00000100:00000010:4.0:1496184949.059982:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000020:00000040:4.0:1496184949.059983:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:4.0:1496184949.059988:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:4.0:1496184949.059989:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:4.0:1496184949.060014:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfea70-0x592def0cfea70 00000100:00000001:4.0:1496184949.060017:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.070169:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.070172:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184949.081187:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.081192:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.081192:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff8810105bae00 00000100:00000001:3.0:1496184949.092184:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:4.0:1496184949.092246:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:4.0:1496184949.092264:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184949.092267:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:4.0:1496184949.092272:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:4.0:1496184949.092273:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:4.0:1496184949.092276:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:4.0:1496184949.092277:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:4.0:1496184949.092310:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:4.0:1496184949.092312:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:4.0:1496184949.092314:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:4.0:1496184949.092315:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:4.0:1496184949.092316:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:4.0:1496184949.092317:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184949.092318:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:4.0:1496184949.092320:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:4.0:1496184949.092321:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:4.0:1496184949.092321:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:4.0:1496184949.092323:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:4.0:1496184949.092324:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff88100bbd7000. 00000100:00000010:4.0:1496184949.092326:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff8810105bae00. 00000100:00000001:4.0:1496184949.092327:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:4.0:1496184949.092328:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:4.0:1496184949.092329:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff881011372c00. 02000000:00000001:4.0:1496184949.092330:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:4.0:1496184949.092332:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:4.0:1496184949.092334:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:4.0:1496184949.092337:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:4.0:1496184949.092342:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:4.0:1496184949.092343:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:4.0:1496184949.092344:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:4.0:1496184949.092345:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:4.0:1496184949.092346:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:4.0:1496184949.092348:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:4.0:1496184949.092354:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 32489us (32636us total) trans 0 rc -110/-999 00000100:00100000:4.0:1496184949.092360:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57661 00000100:00000040:4.0:1496184949.092361:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:4.0:1496184949.092363:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:4.0:1496184949.092364:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:4.0:1496184949.092366:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026079450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:4.0:1496184949.092372:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:4.0:1496184949.092373:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026079450 with x1568861102817424 ext(0->1048575) 00010000:00000001:4.0:1496184949.092374:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:4.0:1496184949.092375:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:4.0:1496184949.092376:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184949.092378:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:4.0:1496184949.092379:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:4.0:1496184949.092380:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:4.0:1496184949.092381:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:4.0:1496184949.092381:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026079450. 00002000:00000001:4.0:1496184949.092383:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:4.0:1496184949.092384:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:4.0:1496184949.092385:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed3ff00. 00000020:00000010:4.0:1496184949.092387:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88103ee07a80. 00000020:00000010:4.0:1496184949.092389:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105ba000. 00000020:00000040:4.0:1496184949.092390:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:4.0:1496184949.092392:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:4.0:1496184949.092394:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:4.0:1496184949.092399:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:4.0:1496184949.092401:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.102902:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.102904:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.102907:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c2050 x1568861104106112 msgsize 520 00000100:00100000:6.0:1496184949.102911:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.102913:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.102916:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.103240:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.103242:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.103244:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e688800. 02000000:00000001:7.0:1496184949.103247:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.103247:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.103656:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.103657:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.103660:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.103665:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.103666:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.103669:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.105598:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.105601:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5800 00000100:00000001:3.0:1496184949.105610:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.105616:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.105617:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5800 00000100:00000001:3.0:1496184949.105624:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.133998:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.134000:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.134002:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1850 x1568861104106144 msgsize 520 00000100:00100000:6.0:1496184949.134007:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.134008:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.134010:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.134357:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.134359:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.134361:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68f000. 02000000:00000001:6.0:1496184949.134364:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.134364:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.134791:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.134793:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.134795:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026078c50 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.134800:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.134802:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.134804:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.136533:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.136548:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.136554:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.136558:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.136559:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.136564:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.164910:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.164912:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.164914:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0450 x1568861104106176 msgsize 520 00000100:00100000:6.0:1496184949.164919:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.164920:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.164931:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.165272:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.165274:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.165276:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68d400. 02000000:00000001:7.0:1496184949.165279:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.165280:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.165687:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.165689:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.165692:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.165697:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.165698:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.165700:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.167404:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.167418:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.167424:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.167429:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.167429:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.167434:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184949.195884:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.195885:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.195888:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1450 x1568861104106208 msgsize 520 00000100:00100000:7.0:1496184949.195913:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.195914:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.195916:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.196310:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.196313:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.196315:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68e800. 02000000:00000001:6.0:1496184949.196320:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.196321:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.196745:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.196747:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.196749:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026077c50 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.196754:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.196755:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.196763:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.198483:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.198486:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.209624:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.209629:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.209630:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.220629:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184949.231570:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.231572:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.231574:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0850 x1568861104106240 msgsize 520 00000100:00100000:7.0:1496184949.231579:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.231580:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.231583:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.231944:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.231947:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.231950:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68c400. 02000000:00000001:6.0:1496184949.231954:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.231955:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.232389:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.232391:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.232393:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026079050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.232398:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.232399:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.232401:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.234100:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.234103:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.245166:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.245171:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.245172:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.256208:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184949.256277:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184949.256279:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184949.256282:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b450 x1568861104106272 msgsize 520 00000100:00100000:1.0:1496184949.256289:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.256290:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.256295:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:2.0:1496184949.256301:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184949.256305:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184949.256309:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:2.0:1496184949.256312:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:2.0:1496184949.256314:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:2.0:1496184949.256316:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1496184949.256320:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1496184949.256324:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106272 00000020:00000001:2.0:1496184949.256327:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:2.0:1496184949.256329:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:2.0:1496184949.256331:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:2.0:1496184949.256334:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:2.0:1496184949.256336:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:2.0:1496184949.256338:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:2.0:1496184949.256342:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.256346:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:2.0:1496184949.256349:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100e966e00. 00000020:00000010:2.0:1496184949.256353:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff881045a05700. 00000020:00000010:2.0:1496184949.256356:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881010ac6600. 00000100:00000001:2.0:1496184949.256359:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:2.0:1496184949.256360:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:2.0:1496184949.256362:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.256367:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:2.0:1496184949.256372:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8b450 x1568861104106272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.256378:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:2.0:1496184949.256379:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256385:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:2.0:1496184949.256389:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.256390:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:2.0:1496184949.256391:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184949.256392:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184949.256393:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184949.256395:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:2.0:1496184949.256398:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6103 00000100:00000040:2.0:1496184949.256400:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.256402:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736393296 : -131906973158320 : ffff880803b8b450) 00000100:00000040:2.0:1496184949.256405:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b450 x1568861104106272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.256410:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.256411:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:2.0:1496184949.256413:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106272:12345-10.37.248.124@o2ib1:8 00000100:00000200:2.0:1496184949.256416:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106272 00000020:00000001:2.0:1496184949.256418:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:2.0:1496184949.256422:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:2.0:1496184949.256430:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:2.0:1496184949.256435:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:2.0:1496184949.256437:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:2.0:1496184949.256439:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88081f7c2c00. 02000000:00000001:2.0:1496184949.256441:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256443:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.256449:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:2.0:1496184949.256452:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:2.0:1496184949.256454:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:2.0:1496184949.256463:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:2.0:1496184949.256465:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:2.0:1496184949.256468:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:2.0:1496184949.256472:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:2.0:1496184949.256475:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:2.0:1496184949.256477:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:2.0:1496184949.256479:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:2.0:1496184949.256481:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:2.0:1496184949.256483:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:2.0:1496184949.256485:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:2.0:1496184949.256486:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:2.0:1496184949.256488:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:2.0:1496184949.256490:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:2.0:1496184949.256492:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:2.0:1496184949.256493:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:2.0:1496184949.256495:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:2.0:1496184949.256500:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:2.0:1496184949.256501:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:2.0:1496184949.256503:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:2.0:1496184949.256505:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:2.0:1496184949.256506:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:2.0:1496184949.256509:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:2.0:1496184949.256511:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:2.0:1496184949.256512:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:2.0:1496184949.256513:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:2.0:1496184949.256517:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:2.0:1496184949.256519:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:2.0:1496184949.256521:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.256522:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:2.0:1496184949.256524:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:2.0:1496184949.256526:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:2.0:1496184949.256528:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.256529:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.256530:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.256532:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184949.256534:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.256535:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184949.256536:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.256538:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.256540:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.256542:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256543:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:2.0:1496184949.256547:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:2.0:1496184949.256549:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184949.256552:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:2.0:1496184949.256554:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.256556:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.256557:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256558:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.256560:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.256561:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.256563:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1496184949.256566:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1496184949.256568:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256569:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:2.0:1496184949.256572:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:2.0:1496184949.256573:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1496184949.256575:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:2.0:1496184949.256577:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:2.0:1496184949.256578:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:2.0:1496184949.256588:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:2.0:1496184949.256590:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.256592:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.256593:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:2.0:1496184949.256595:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.256598:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:2.0:1496184949.256600:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:2.0:1496184949.256601:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:2.0:1496184949.256602:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1496184949.256604:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:2.0:1496184949.256605:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:2.0:1496184949.256607:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.256608:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:2.0:1496184949.256611:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:2.0:1496184949.256613:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106272 00010000:00000001:2.0:1496184949.256615:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:2.0:1496184949.256617:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8b450 x1568861104106272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:2.0:1496184949.256624:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:2.0:1496184949.256625:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1496184949.256627:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:2.0:1496184949.256629:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:2.0:1496184949.256632:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.256633:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.256634:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.256636:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:2.0:1496184949.256637:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256638:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:2.0:1496184949.256640:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:2.0:1496184949.256644:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106272, offset 0 00000100:00000001:2.0:1496184949.256660:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.256662:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.256663:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.256665:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.256666:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:2.0:1496184949.256667:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.256668:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:2.0:1496184949.256670:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b450 x1568861104106272/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184949.256682:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106272:12345-10.37.248.124@o2ib1:8 Request procesed in 264us (392us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184949.256687:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6103 00000100:00000040:2.0:1496184949.256689:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:2.0:1496184949.256691:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:2.0:1496184949.256692:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:2.0:1496184949.256695:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff881045a05700. 00000020:00000010:2.0:1496184949.256697:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881010ac6600. 00000020:00000010:2.0:1496184949.256699:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100e966e00. 00000020:00000040:2.0:1496184949.256701:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:2.0:1496184949.256703:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:2.0:1496184949.256706:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:2.0:1496184949.256750:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.256751:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.257027:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.257029:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.257031:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88081f7c2c00. 02000000:00000001:6.0:1496184949.257034:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.257035:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.257498:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.257500:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.257503:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600f450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.257508:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.257509:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.257511:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.270026:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.270028:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00000001:6.0:1496184949.274946:0:13696:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184949.274948:0:13696:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184949.274952:0:13696:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184949.274952:0:13696:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.274954:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184949.274956:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.274957:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184949.274958:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184949.277955:0:13694:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:7.0:1496184949.277957:0:13694:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:7.0:1496184949.277961:0:13694:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:7.0:1496184949.277962:0:13694:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184949.277963:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184949.277965:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184949.277966:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184949.277967:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184949.280946:0:13722:0:(ldlm_pool.c:339:ldlm_srv_pool_recalc()) Process entered 00010000:00000001:6.0:1496184949.280948:0:13722:0:(ldlm_pool.c:371:ldlm_srv_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184949.280955:0:13722:0:(ldlm_pool.c:478:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:6.0:1496184949.280956:0:13722:0:(ldlm_pool.c:482:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.280991:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184949.280993:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.280995:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184949.280996:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184949.281011:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184949.281013:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184949.281015:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184949.281016:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.281029:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.281033:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.281034:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000001:6.0:1496184949.290949:0:13693:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184949.290951:0:13693:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184949.290954:0:13693:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184949.290955:0:13693:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.290957:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184949.290958:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184949.290959:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184949.290960:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.292011:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184949.303261:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184949.303263:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184949.303266:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a050 x1568861104106304 msgsize 520 00000100:00100000:1.0:1496184949.303272:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.303273:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.303276:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:2.0:1496184949.303282:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184949.303285:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184949.303287:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:2.0:1496184949.303290:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:2.0:1496184949.303292:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:2.0:1496184949.303294:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1496184949.303298:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1496184949.303301:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106304 00000020:00000001:2.0:1496184949.303303:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:2.0:1496184949.303307:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:2.0:1496184949.303308:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:2.0:1496184949.303310:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:2.0:1496184949.303312:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:2.0:1496184949.303314:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:2.0:1496184949.303318:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.303318:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:2.0:1496184949.303321:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100e966e00. 00000020:00000010:2.0:1496184949.303324:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff881045a05480. 00000020:00000010:2.0:1496184949.303326:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff881010ac6600. 00000100:00000001:2.0:1496184949.303328:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:2.0:1496184949.303329:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:2.0:1496184949.303330:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.303334:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:2.0:1496184949.303338:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8a050 x1568861104106304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.303343:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:2.0:1496184949.303345:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303348:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:2.0:1496184949.303350:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.303352:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:2.0:1496184949.303353:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184949.303353:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184949.303355:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184949.303356:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:2.0:1496184949.303358:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6104 00000100:00000040:2.0:1496184949.303360:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.303362:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736388176 : -131906973163440 : ffff880803b8a050) 00000100:00000040:2.0:1496184949.303364:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a050 x1568861104106304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.303369:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:2.0:1496184949.303370:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:2.0:1496184949.303372:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106304:12345-10.37.248.124@o2ib1:8 00000100:00000200:2.0:1496184949.303376:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106304 00000020:00000001:2.0:1496184949.303378:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:2.0:1496184949.303381:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:2.0:1496184949.303388:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:2.0:1496184949.303392:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:2.0:1496184949.303393:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:2.0:1496184949.303396:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88081f7c4400. 02000000:00000001:2.0:1496184949.303397:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303399:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.303403:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:2.0:1496184949.303405:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:2.0:1496184949.303407:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:2.0:1496184949.303409:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:2.0:1496184949.303411:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:2.0:1496184949.303413:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:2.0:1496184949.303416:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:2.0:1496184949.303418:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:2.0:1496184949.303420:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:2.0:1496184949.303422:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:2.0:1496184949.303423:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:2.0:1496184949.303424:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:2.0:1496184949.303426:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:2.0:1496184949.303427:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:2.0:1496184949.303429:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:2.0:1496184949.303430:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:2.0:1496184949.303431:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:2.0:1496184949.303433:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:2.0:1496184949.303434:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:2.0:1496184949.303438:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:2.0:1496184949.303444:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:2.0:1496184949.303445:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:2.0:1496184949.303449:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:2.0:1496184949.303450:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:2.0:1496184949.303452:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:2.0:1496184949.303454:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:2.0:1496184949.303455:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:2.0:1496184949.303456:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:2.0:1496184949.303458:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:2.0:1496184949.303459:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:2.0:1496184949.303461:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.303462:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:2.0:1496184949.303464:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:2.0:1496184949.303466:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:2.0:1496184949.303467:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:2.0:1496184949.303468:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.303469:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.303470:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:2.0:1496184949.303472:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.303473:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:2.0:1496184949.303474:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184949.303476:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.303478:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.303479:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303480:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:2.0:1496184949.303484:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:2.0:1496184949.303486:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:2.0:1496184949.303488:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:2.0:1496184949.303490:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.303491:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.303492:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303493:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.303495:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.303496:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.303498:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:2.0:1496184949.303500:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1496184949.303501:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303503:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:2.0:1496184949.303504:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:2.0:1496184949.303505:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:2.0:1496184949.303506:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:2.0:1496184949.303508:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:2.0:1496184949.303509:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:2.0:1496184949.303510:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:2.0:1496184949.303512:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1496184949.303513:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:2.0:1496184949.303515:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:2.0:1496184949.303516:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.303519:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:2.0:1496184949.303520:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:2.0:1496184949.303522:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:2.0:1496184949.303523:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:2.0:1496184949.303524:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:2.0:1496184949.303525:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:2.0:1496184949.303527:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.303528:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:2.0:1496184949.303530:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:2.0:1496184949.303531:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106304 00010000:00000001:2.0:1496184949.303533:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:2.0:1496184949.303536:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8a050 x1568861104106304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:2.0:1496184949.303541:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:2.0:1496184949.303542:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1496184949.303543:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:2.0:1496184949.303545:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:2.0:1496184949.303547:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:2.0:1496184949.303549:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.303550:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:2.0:1496184949.303552:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:2.0:1496184949.303553:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303554:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:2.0:1496184949.303556:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:2.0:1496184949.303559:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106304, offset 0 00000100:00000001:2.0:1496184949.303577:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.303579:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:2.0:1496184949.303580:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:2.0:1496184949.303581:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1496184949.303582:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:2.0:1496184949.303583:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.303585:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:2.0:1496184949.303586:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a050 x1568861104106304/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:2.0:1496184949.303592:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106304:12345-10.37.248.124@o2ib1:8 Request procesed in 221us (324us total) trans 0 rc 0/0 00000100:00100000:2.0:1496184949.303596:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6104 00000100:00000040:2.0:1496184949.303598:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:2.0:1496184949.303600:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:2.0:1496184949.303601:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:2.0:1496184949.303603:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff881045a05480. 00000020:00000010:2.0:1496184949.303604:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff881010ac6600. 00000020:00000010:2.0:1496184949.303606:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100e966e00. 00000020:00000040:2.0:1496184949.303608:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:2.0:1496184949.303609:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:2.0:1496184949.303611:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:2.0:1496184949.303615:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184949.303616:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.303633:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.303635:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.303637:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88081f7c4400. 02000000:00000001:6.0:1496184949.303640:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.303641:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.304098:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.304100:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.304103:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026078050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.304108:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.304109:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.304112:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.306387:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.306389:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.306395:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.306399:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.306400:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7600 00000100:00000001:3.0:1496184949.306405:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.334902:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.334904:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.334907:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c1050 x1568861104106336 msgsize 520 00000100:00100000:6.0:1496184949.334912:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.334913:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.334915:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.335265:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.335267:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.335269:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68b800. 02000000:00000001:7.0:1496184949.335272:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.335273:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.335772:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.335774:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.335776:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026079850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.335781:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.335782:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.335784:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.337682:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.337684:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.337690:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.337694:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.337695:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.337700:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.341956:0:13691:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184949.341956:0:13689:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1496184949.341958:0:13691:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.341958:0:13689:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.341964:0:13691:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184949.341964:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184949.341965:0:13691:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.341965:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.341967:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184949.341967:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184949.341970:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.341970:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.341971:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184949.341972:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.341973:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184949.341974:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342902:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) d86c02e9-1939-02b8-4331-d3612fcf0106->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1496184949.342907:0:13697:0:(pinger.c:97:ptlrpc_ping()) Process entered 00000100:00000010:2.0:1496184949.342911:0:13697:0:(client.c:493:ptlrpc_request_cache_alloc()) slab-alloced 'req': 760 at ffff88080f768600. 00000100:00000001:0.0:1496184949.342914:0:13690:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000020:00000040:2.0:1496184949.342917:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103ef23800 refcount=4 obd=MGC10.37.248.196@o2ib1 00000100:00000001:0.0:1496184949.342917:0:13690:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.342932:0:13697:0:(client.c:700:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1496184949.342933:0:13697:0:(sec.c:434:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1496184949.342936:0:13697:0:(sec.c:452:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.342936:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1496184949.342937:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.342938:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 02000000:00000010:2.0:1496184949.342939:0:13697:0:(sec_null.c:161:null_alloc_reqbuf()) kmalloced 'req->rq_reqbuf': 256 at ffff880822230900. 00000100:00000001:0.0:1496184949.342940:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.342941:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184949.342942:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.342943:0:13697:0:(client.c:774:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342947:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging d86c02e9-1939-02b8-4331-d3612fcf0106->MGS req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000020:00000001:2.0:1496184949.342953:0:13697:0:(class_obd.c:128:lustre_get_jobid()) Process entered 00000020:00000001:2.0:1496184949.342955:0:13697:0:(class_obd.c:145:lustre_get_jobid()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1496184949.342957:0:13697:0:(class_obd.c:181:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342960:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f768600] to pc [ptlrpcd_00_03:3] req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.342968:0:13697:0:(pinger.c:111:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342969:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1496184949.342971:0:13697:0:(pinger.c:97:ptlrpc_ping()) Process entered 00000100:00000010:2.0:1496184949.342972:0:13697:0:(client.c:493:ptlrpc_request_cache_alloc()) slab-alloced 'req': 760 at ffff88080f769800. 00000020:00000040:2.0:1496184949.342973:0:13697:0:(genops.c:1023:class_import_get()) import ffff881043bd0000 refcount=3 obd=lustre-MDT0000-lwp-OST0000 00000100:00000001:2.0:1496184949.342975:0:13697:0:(client.c:700:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1496184949.342975:0:13697:0:(sec.c:434:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1496184949.342976:0:13697:0:(sec.c:452:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:2.0:1496184949.342977:0:13697:0:(sec_null.c:161:null_alloc_reqbuf()) kmalloced 'req->rq_reqbuf': 256 at ffff880822230300. 00000100:00000001:2.0:1496184949.342979:0:13697:0:(client.c:774:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342981:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0000_UUID->lustre-MDT0000_UUID req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000020:00000001:2.0:1496184949.342985:0:13697:0:(class_obd.c:128:lustre_get_jobid()) Process entered 00000020:00000001:2.0:1496184949.342986:0:13697:0:(class_obd.c:145:lustre_get_jobid()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1496184949.342987:0:13697:0:(class_obd.c:181:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342989:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f769800] to pc [ptlrpcd_00_00:0] req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.342995:0:13697:0:(pinger.c:111:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.342996:0:13697:0:(pinger.c:217:ptlrpc_pinger_process_import()) lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1496184949.343000:0:13697:0:(pinger.c:97:ptlrpc_ping()) Process entered 00000100:00000001:3.0:1496184949.343001:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000010:2.0:1496184949.343001:0:13697:0:(client.c:493:ptlrpc_request_cache_alloc()) slab-alloced 'req': 760 at ffff88080f768c00. 00000020:00000040:2.0:1496184949.343002:0:13697:0:(genops.c:1023:class_import_get()) import ffff88103770e800 refcount=3 obd=lustre-MDT0000-lwp-OST0001 00000100:00000001:1.0:1496184949.343002:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343003:0:13697:0:(client.c:700:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1496184949.343004:0:13697:0:(sec.c:434:sptlrpc_req_get_ctx()) Process entered 00000100:00000001:3.0:1496184949.343005:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1496184949.343005:0:13697:0:(sec.c:452:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:2.0:1496184949.343006:0:13697:0:(sec_null.c:161:null_alloc_reqbuf()) kmalloced 'req->rq_reqbuf': 256 at ffff880822230600. 00000100:00000001:1.0:1496184949.343006:0:13692:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1496184949.343007:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343007:0:13697:0:(client.c:774:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343008:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343008:0:13692:0:(client.c:1530:ptlrpc_send_new_req()) Process entered 00000100:00000040:2.0:1496184949.343009:0:13697:0:(pinger.c:108:ptlrpc_ping()) @@@ pinging lustre-MDT0000-lwp-OST0001_UUID->lustre-MDT0000_UUID req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000020:00000001:2.0:1496184949.343013:0:13697:0:(class_obd.c:128:lustre_get_jobid()) Process entered 00000100:00000040:1.0:1496184949.343013:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000020:00000001:2.0:1496184949.343014:0:13697:0:(class_obd.c:145:lustre_get_jobid()) Process leaving via out (rc=0 : 0 : 0x0) 00000020:00000001:2.0:1496184949.343015:0:13697:0:(class_obd.c:181:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.343017:0:13697:0:(ptlrpcd.c:299:ptlrpcd_add_req()) @@@ add req [ffff88080f768c00] to pc [ptlrpcd_00_01:1] req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.343021:0:13697:0:(pinger.c:111:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343021:0:13692:0:(client.c:1153:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1496184949.343022:0:13692:0:(client.c:1204:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184949.343024:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=1448666 : 1448666 : 161ada) 00000100:00000001:0.0:1496184949.343024:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 02000000:00000001:1.0:1496184949.343025:0:13692:0:(sec.c:685:sptlrpc_req_refresh_ctx()) Process entered 00000020:00000001:2.0:1496184949.343026:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=1021464 : 1021464 : f9618) 00000020:00000001:2.0:1496184949.343027:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=1979528 : 1979528 : 1e3488) 02000000:00000001:1.0:1496184949.343027:0:13692:0:(sec.c:712:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343027:0:13690:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000020:00000001:2.0:1496184949.343028:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=2137800 : 2137800 : 209ec8) 00000100:00000001:0.0:1496184949.343028:0:13690:0:(client.c:1530:ptlrpc_send_new_req()) Process entered 00000020:00000001:2.0:1496184949.343029:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=4655903 : 4655903 : 470b1f) 00000100:00100000:1.0:1496184949.343029:0:13692:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:d86c02e9-1939-02b8-4331-d3612fcf0106:13692:1568861159710288:10.37.248.196@o2ib1:400 00000020:00000001:2.0:1496184949.343031:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=143888567 : 143888567 : 89390b7) 00000020:00000001:2.0:1496184949.343032:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=18446744073708119136 : -1432480 : ffffffffffea2460) 00000100:00000001:1.0:1496184949.343032:0:13692:0:(niobuf.c:699:ptl_send_rpc()) Process entered 00000020:00000001:2.0:1496184949.343033:0:13697:0:(lprocfs_status.c:1829:lprocfs_read_helper()) Process leaving (rc=650477 : 650477 : 9eced) 00000100:00000040:0.0:1496184949.343033:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 02000000:00000001:1.0:1496184949.343034:0:13692:0:(sec.c:986:sptlrpc_cli_wrap_request()) Process entered 00000100:00000040:2.0:1496184949.343035:0:13697:0:(pinger.c:289:ptlrpc_pinger_main()) next wakeup in 4999 (4297388370) 02000000:00000001:1.0:1496184949.343036:0:13692:0:(sec.c:1022:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1496184949.343037:0:13692:0:(sec.c:1692:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1496184949.343039:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184949.343039:0:13690:0:(client.c:1153:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1496184949.343040:0:13690:0:(client.c:1204:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343041:0:13689:0:(client.c:1680:ptlrpc_check_set()) Process entered 02000000:00000010:1.0:1496184949.343041:0:13692:0:(sec_null.c:204:null_alloc_repbuf()) kmalloced 'req->rq_repbuf': 512 at ffff88085de87400. 00000100:00000001:3.0:1496184949.343042:0:13689:0:(client.c:1530:ptlrpc_send_new_req()) Process entered 02000000:00000001:1.0:1496184949.343042:0:13692:0:(sec.c:1702:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1496184949.343042:0:13690:0:(sec.c:685:sptlrpc_req_refresh_ctx()) Process entered 02000000:00000001:0.0:1496184949.343043:0:13690:0:(sec.c:712:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.343046:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Rpc" req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1496184949.343046:0:13690:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710320:10.37.248.197@o2ib1:400 00000100:00000001:0.0:1496184949.343048:0:13690:0:(niobuf.c:699:ptl_send_rpc()) Process entered 00000100:00000200:1.0:1496184949.343049:0:13692:0:(niobuf.c:875:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1568861159710288, portal 25 02000000:00000001:0.0:1496184949.343049:0:13690:0:(sec.c:986:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1496184949.343050:0:13692:0:(client.c:2808:ptlrpc_request_addref()) Process entered 02000000:00000001:0.0:1496184949.343050:0:13690:0:(sec.c:1022:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343051:0:13692:0:(client.c:2810:ptlrpc_request_addref()) Process leaving (rc=18446612166933382656 : -131906776168960 : ffff88080f768600) 00000100:00000001:3.0:1496184949.343052:0:13689:0:(client.c:1153:ptlrpc_import_delay_req()) Process entered 02000000:00000001:0.0:1496184949.343052:0:13690:0:(sec.c:1692:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:3.0:1496184949.343053:0:13689:0:(client.c:1204:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:0.0:1496184949.343054:0:13690:0:(sec_null.c:204:null_alloc_repbuf()) kmalloced 'req->rq_repbuf': 512 at ffff8810101cce00. 02000000:00000001:3.0:1496184949.343055:0:13689:0:(sec.c:685:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000040:1.0:1496184949.343055:0:13692:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 02000000:00000001:0.0:1496184949.343055:0:13690:0:(sec.c:1702:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184949.343056:0:13689:0:(sec.c:712:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184949.343058:0:13689:0:(client.c:1620:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710304:10.37.248.197@o2ib1:400 00000100:00000001:1.0:1496184949.343060:0:13692:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000200:0.0:1496184949.343061:0:13690:0:(niobuf.c:875:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1568861159710320, portal 10 00000100:00000040:1.0:1496184949.343062:0:13692:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.196@o2ib1 00000100:00000001:0.0:1496184949.343063:0:13690:0:(client.c:2808:ptlrpc_request_addref()) Process entered 00000100:00000001:0.0:1496184949.343063:0:13690:0:(client.c:2810:ptlrpc_request_addref()) Process leaving (rc=18446612166933384192 : -131906776167424 : ffff88080f768c00) 00000100:00000001:3.0:1496184949.343064:0:13689:0:(niobuf.c:699:ptl_send_rpc()) Process entered 00000100:00000200:1.0:1496184949.343064:0:13692:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1568861159710288, offset 0 02000000:00000001:3.0:1496184949.343066:0:13689:0:(sec.c:986:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:3.0:1496184949.343067:0:13689:0:(sec.c:1022:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1496184949.343067:0:13690:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 02000000:00000001:3.0:1496184949.343068:0:13689:0:(sec.c:1692:sptlrpc_cli_alloc_repbuf()) Process entered 02000000:00000010:3.0:1496184949.343070:0:13689:0:(sec_null.c:204:null_alloc_repbuf()) kmalloced 'req->rq_repbuf': 512 at ffff8810105bdc00. 02000000:00000001:3.0:1496184949.343071:0:13689:0:(sec.c:1702:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343072:0:13690:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:0.0:1496184949.343073:0:13690:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000200:0.0:1496184949.343075:0:13690:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1568861159710320, offset 0 00000100:00000200:3.0:1496184949.343076:0:13689:0:(niobuf.c:875:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1568861159710304, portal 10 00000100:00000001:1.0:1496184949.343077:0:13692:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343078:0:13689:0:(client.c:2808:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1496184949.343079:0:13689:0:(client.c:2810:ptlrpc_request_addref()) Process leaving (rc=18446612166933387264 : -131906776164352 : ffff88080f769800) 00000100:00000001:1.0:1496184949.343079:0:13692:0:(niobuf.c:904:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1496184949.343080:0:13692:0:(client.c:1640:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343083:0:13692:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343084:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.343087:0:13692:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1496184949.343087:0:13690:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343088:0:13692:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=6 : 6 : 6) 00000100:00000001:0.0:1496184949.343088:0:13690:0:(niobuf.c:904:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:1.0:1496184949.343089:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184949.343090:0:13692:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1496184949.343090:0:13690:0:(client.c:1640:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.343091:0:13689:0:(niobuf.c:896:ptl_send_rpc()) @@@ send flg=0 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:1.0:1496184949.343091:0:13692:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343091:0:13690:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343092:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.343094:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343095:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184949.343095:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184949.343096:0:13689:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000001:1.0:1496184949.343096:0:13692:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1496184949.343097:0:13692:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.343098:0:13689:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000001:1.0:1496184949.343098:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343098:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=6 : 6 : 6) 00000100:00000001:0.0:1496184949.343099:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000200:3.0:1496184949.343100:0:13689:0:(niobuf.c:85:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1568861159710304, offset 0 00000100:00000001:0.0:1496184949.343100:0:13690:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1496184949.343101:0:13690:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343102:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343103:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343104:0:6132:0:(events.c:52:request_out_callback()) Process entered 00000100:00000001:0.0:1496184949.343104:0:13690:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1496184949.343105:0:13690:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:2.0:1496184949.343106:0:6132:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184949.343106:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343110:0:13689:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343111:0:13689:0:(niobuf.c:904:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1496184949.343111:0:6132:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00000100:00000001:3.0:1496184949.343113:0:13689:0:(client.c:1640:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.343113:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.343114:0:13689:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343115:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.343117:0:6132:0:(client.c:2531:__ptlrpc_req_finished()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343118:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184949.343119:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=6 : 6 : 6) 00000100:00000001:2.0:1496184949.343119:0:6132:0:(events.c:81:request_out_callback()) Process leaving 00000100:00000001:3.0:1496184949.343120:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184949.343121:0:13689:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1496184949.343121:0:6131:0:(events.c:52:request_out_callback()) Process entered 00000100:00000001:3.0:1496184949.343122:0:13689:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343123:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:1.0:1496184949.343123:0:6131:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.343124:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184949.343125:0:13689:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:3.0:1496184949.343125:0:13689:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.343126:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343128:0:6131:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00000100:00000040:1.0:1496184949.343130:0:6131:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:1.0:1496184949.343134:0:6131:0:(client.c:2531:__ptlrpc_req_finished()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343135:0:6131:0:(events.c:81:request_out_callback()) Process leaving 00000100:00000001:2.0:1496184949.343137:0:6132:0:(events.c:52:request_out_callback()) Process entered 00000100:00000200:2.0:1496184949.343138:0:6132:0:(events.c:57:request_out_callback()) @@@ type 5, status 0 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.343142:0:6132:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00000100:00000040:2.0:1496184949.343144:0:6132:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 2 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.343147:0:6132:0:(client.c:2531:__ptlrpc_req_finished()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343148:0:6132:0:(events.c:81:request_out_callback()) Process leaving 00000100:00000001:2.0:1496184949.343366:0:6131:0:(events.c:91:reply_in_callback()) Process entered 00000100:00000200:2.0:1496184949.343369:0:6131:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184949.343376:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.343385:0:6131:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:2.0:1496184949.343402:0:6131:0:(events.c:91:reply_in_callback()) Process entered 00000100:00000200:2.0:1496184949.343404:0:6131:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000040:2.0:1496184949.343408:0:6131:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:2.0:1496184949.343413:0:6131:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.343417:0:6132:0:(events.c:91:reply_in_callback()) Process entered 00000100:00000200:3.0:1496184949.343422:0:6132:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184949.343424:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184949.343427:0:13690:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000040:3.0:1496184949.343429:0:6132:0:(events.c:161:reply_in_callback()) @@@ reply in flags=0 mlen=192 offset=192 replen=224 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.343438:0:6132:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:2.0:1496184949.343440:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343442:0:13692:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1496184949.343443:0:13690:0:(events.c:91:reply_in_callback()) Process entered 00000100:00000001:1.0:1496184949.343445:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000200:0.0:1496184949.343447:0:13690:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:1.0:1496184949.343448:0:13689:0:(client.c:1680:ptlrpc_check_set()) Process entered 00000100:00000200:0.0:1496184949.343454:0:13690:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184949.343458:0:13690:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:2.0:1496184949.343461:0:13692:0:(events.c:91:reply_in_callback()) Process entered 00000100:00000001:0.0:1496184949.343461:0:13690:0:(client.c:2587:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:2.0:1496184949.343463:0:13692:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184949.343463:0:13690:0:(client.c:1337:after_reply()) Process entered 02000000:00000001:0.0:1496184949.343465:0:13690:0:(sec.c:1029:do_cli_unwrap_reply()) Process entered 00000100:00000001:0.0:1496184949.343466:0:13690:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:0.0:1496184949.343467:0:13690:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:2.0:1496184949.343468:0:13692:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 02000000:00000001:0.0:1496184949.343469:0:13690:0:(sec.c:1083:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343471:0:13692:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:2.0:1496184949.343473:0:13692:0:(client.c:2587:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00001000:0.0:1496184949.343473:0:13690:0:(import.c:1678:at_measured()) add 1 to ffff88103770ec00 time=98 v=1 (1 1 1 1) 00000100:00000001:2.0:1496184949.343474:0:13692:0:(client.c:1337:after_reply()) Process entered 00000100:00000001:1.0:1496184949.343474:0:13689:0:(events.c:91:reply_in_callback()) Process entered 02000000:00000001:2.0:1496184949.343475:0:13692:0:(sec.c:1029:do_cli_unwrap_reply()) Process entered 00000100:00000001:2.0:1496184949.343476:0:13692:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00001000:0.0:1496184949.343476:0:13690:0:(import.c:1678:at_measured()) add 1 to ffff88103770ebc8 time=56 v=1 (1 1 1 1) 00000100:00000001:2.0:1496184949.343477:0:13692:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343477:0:13690:0:(client.c:1251:ptlrpc_check_status()) Process entered 00000100:00000200:1.0:1496184949.343478:0:13689:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000001:0.0:1496184949.343478:0:13690:0:(client.c:1277:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1496184949.343479:0:13692:0:(sec.c:1083:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1496184949.343480:0:13690:0:(ldlm_request.c:1308:ldlm_cli_update_pool()) Process entered 00000100:00001000:2.0:1496184949.343481:0:13692:0:(import.c:1678:at_measured()) add 1 to ffff88103ef23c00 time=105 v=1 (1 1 1 1) 00010000:00000001:0.0:1496184949.343482:0:13690:0:(ldlm_request.c:1346:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1496184949.343483:0:13692:0:(import.c:1678:at_measured()) add 1 to ffff88103ef23bc8 time=105 v=1 (1 1 1 1) 00000100:00000001:2.0:1496184949.343484:0:13692:0:(client.c:1251:ptlrpc_check_status()) Process entered 00000100:00000001:0.0:1496184949.343484:0:13690:0:(client.c:2675:ptlrpc_free_committed()) Process entered 00000100:00000001:2.0:1496184949.343485:0:13692:0:(client.c:1277:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:1.0:1496184949.343485:0:13689:0:(events.c:114:reply_in_callback()) @@@ unlink req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/224 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/ffffffff rc 0/-1 00000100:00000040:0.0:1496184949.343485:0:13690:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0001: skip recheck: last_committed 0 00010000:00000001:2.0:1496184949.343486:0:13692:0:(ldlm_request.c:1308:ldlm_cli_update_pool()) Process entered 00000100:00000001:0.0:1496184949.343486:0:13690:0:(client.c:2684:ptlrpc_free_committed()) Process leaving 00010000:00000001:2.0:1496184949.343487:0:13692:0:(ldlm_request.c:1315:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343487:0:13690:0:(client.c:1517:after_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343488:0:13692:0:(client.c:1517:after_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1496184949.343489:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000040:2.0:1496184949.343490:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/0 rc 0/0 00000100:00000001:1.0:1496184949.343490:0:13689:0:(events.c:171:reply_in_callback()) Process leaving 00000100:00000001:1.0:1496184949.343492:0:13689:0:(client.c:2587:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.343493:0:13689:0:(client.c:1337:after_reply()) Process entered 00000100:00000001:2.0:1496184949.343494:0:13692:0:(client.c:1990:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1496184949.343494:0:13690:0:(client.c:1990:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 02000000:00000001:1.0:1496184949.343495:0:13689:0:(sec.c:1029:do_cli_unwrap_reply()) Process entered 00000100:00000001:2.0:1496184949.343496:0:13692:0:(niobuf.c:436:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1496184949.343496:0:13689:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:0.0:1496184949.343496:0:13690:0:(niobuf.c:436:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:1.0:1496184949.343497:0:13689:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343498:0:13692:0:(niobuf.c:446:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:1.0:1496184949.343498:0:13689:0:(sec.c:1083:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1496184949.343500:0:13692:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00000001:0.0:1496184949.343500:0:13690:0:(niobuf.c:446:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00001000:1.0:1496184949.343501:0:13689:0:(import.c:1678:at_measured()) add 1 to ffff881043bd0400 time=104 v=1 (1 1 1 1) 00000100:00000040:0.0:1496184949.343502:0:13690:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Interpret:RN/0/0 rc 0/0 00000100:00001000:1.0:1496184949.343503:0:13689:0:(import.c:1678:at_measured()) add 1 to ffff881043bd03c8 time=56 v=1 (1 1 1 1) 00000100:00100000:2.0:1496184949.343505:0:13692:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_03:d86c02e9-1939-02b8-4331-d3612fcf0106:13692:1568861159710288:10.37.248.196@o2ib1:400 00000100:00000001:1.0:1496184949.343505:0:13689:0:(client.c:1251:ptlrpc_check_status()) Process entered 00000100:00000001:1.0:1496184949.343506:0:13689:0:(client.c:1277:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343507:0:13692:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00010000:00000001:1.0:1496184949.343507:0:13689:0:(ldlm_request.c:1308:ldlm_cli_update_pool()) Process entered 00000100:00100000:0.0:1496184949.343507:0:13690:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_01:lustre-MDT0000-lwp-OST0001_UUID:13690:1568861159710320:10.37.248.197@o2ib1:400 00000100:00000001:2.0:1496184949.343508:0:13692:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00010000:00000001:1.0:1496184949.343508:0:13689:0:(ldlm_request.c:1346:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343509:0:13690:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:2.0:1496184949.343510:0:13692:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f768600 x1568861159710288/t0(0) o400->MGC10.37.248.196@o2ib1@10.37.248.196@o2ib1:26/25 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000001:1.0:1496184949.343510:0:13689:0:(client.c:2675:ptlrpc_free_committed()) Process entered 00000100:00000001:0.0:1496184949.343510:0:13690:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00000100:00000040:1.0:1496184949.343511:0:13689:0:(client.c:2683:ptlrpc_free_committed()) lustre-MDT0000-lwp-OST0000: skip recheck: last_committed 0 00000100:00000001:1.0:1496184949.343512:0:13689:0:(client.c:2684:ptlrpc_free_committed()) Process leaving 00000100:00000001:1.0:1496184949.343512:0:13689:0:(client.c:1517:after_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1496184949.343512:0:13690:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f768c00 x1568861159710320/t0(0) o400->lustre-MDT0000-lwp-OST0001@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000001:2.0:1496184949.343514:0:13692:0:(client.c:2438:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1496184949.343515:0:13692:0:(sec.c:1713:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:0.0:1496184949.343516:0:13690:0:(client.c:2438:__ptlrpc_free_req()) Process entered 02000000:00000010:2.0:1496184949.343517:0:13692:0:(sec_null.c:218:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at ffff88085de87400. 00000100:00000040:1.0:1496184949.343517:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret" req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Rpc:RN/0/0 rc 0/0 02000000:00000001:0.0:1496184949.343517:0:13690:0:(sec.c:1713:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000010:0.0:1496184949.343518:0:13690:0:(sec_null.c:218:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at ffff8810101cce00. 02000000:00000001:2.0:1496184949.343519:0:13692:0:(sec.c:1727:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1496184949.343520:0:13692:0:(genops.c:1030:class_import_put()) Process entered 02000000:00000001:0.0:1496184949.343520:0:13690:0:(sec.c:1727:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000040:2.0:1496184949.343521:0:13692:0:(genops.c:1037:class_import_put()) import ffff88103ef23800 refcount=3 obd=MGC10.37.248.196@o2ib1 00000100:00000001:1.0:1496184949.343521:0:13689:0:(client.c:1990:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000020:00000001:0.0:1496184949.343521:0:13690:0:(genops.c:1030:class_import_put()) Process entered 00000020:00000040:0.0:1496184949.343521:0:13690:0:(genops.c:1037:class_import_put()) import ffff88103770e800 refcount=2 obd=lustre-MDT0000-lwp-OST0001 00000020:00000001:2.0:1496184949.343522:0:13692:0:(genops.c:1046:class_import_put()) Process leaving 02000000:00000010:2.0:1496184949.343523:0:13692:0:(sec_null.c:188:null_free_reqbuf()) kfreed 'req->rq_reqbuf': 256 at ffff880822230900. 00000100:00000001:1.0:1496184949.343523:0:13689:0:(niobuf.c:436:ptlrpc_unregister_bulk()) Process entered 00000020:00000001:0.0:1496184949.343523:0:13690:0:(genops.c:1046:class_import_put()) Process leaving 00000100:00000001:1.0:1496184949.343524:0:13689:0:(niobuf.c:446:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 02000000:00000010:0.0:1496184949.343524:0:13690:0:(sec_null.c:188:null_free_reqbuf()) kfreed 'req->rq_reqbuf': 256 at ffff880822230600. 02000000:00000001:2.0:1496184949.343525:0:13692:0:(sec.c:466:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1496184949.343525:0:13690:0:(sec.c:466:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1496184949.343526:0:13692:0:(sec.c:482:sptlrpc_req_put_ctx()) Process leaving 00000100:00000040:1.0:1496184949.343526:0:13689:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Interpret:RN/0/0 rc 0/0 02000000:00000001:0.0:1496184949.343526:0:13690:0:(sec.c:482:sptlrpc_req_put_ctx()) Process leaving 00000100:00000010:2.0:1496184949.343527:0:13692:0:(client.c:499:ptlrpc_request_cache_free()) slab-freed '(req)': 760 at ffff88080f768600. 00000100:00000010:0.0:1496184949.343527:0:13690:0:(client.c:499:ptlrpc_request_cache_free()) slab-freed '(req)': 760 at ffff88080f768c00. 00000100:00000001:2.0:1496184949.343528:0:13692:0:(client.c:2490:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1496184949.343529:0:13692:0:(client.c:2528:__ptlrpc_req_finished()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184949.343529:0:13690:0:(client.c:2490:__ptlrpc_free_req()) Process leaving 00000100:00000001:0.0:1496184949.343529:0:13690:0:(client.c:2528:__ptlrpc_req_finished()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.343530:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184949.343530:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:1.0:1496184949.343531:0:13689:0:(client.c:2045:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_00_00:lustre-MDT0000-lwp-OST0000_UUID:13689:1568861159710304:10.37.248.197@o2ib1:400 00000100:00000001:2.0:1496184949.343532:0:13692:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184949.343533:0:13689:0:(client.c:2089:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184949.343533:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184949.343534:0:13692:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343534:0:13689:0:(client.c:2512:__ptlrpc_req_finished()) Process entered 00000100:00000001:0.0:1496184949.343534:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343535:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000040:1.0:1496184949.343535:0:13689:0:(client.c:2524:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff88080f769800 x1568861159710304/t0(0) o400->lustre-MDT0000-lwp-OST0000@10.37.248.197@o2ib1:12/10 lens 224/192 e 0 to 0 dl 1496184956 ref 1 fl Complete:RN/0/0 rc 0/0 00000100:00000001:0.0:1496184949.343535:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343536:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184949.343537:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.343538:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184949.343538:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.343539:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343539:0:13689:0:(client.c:2438:__ptlrpc_free_req()) Process entered 02000000:00000001:1.0:1496184949.343540:0:13689:0:(sec.c:1713:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:0.0:1496184949.343540:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:1.0:1496184949.343541:0:13689:0:(sec_null.c:218:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at ffff8810105bdc00. 02000000:00000001:1.0:1496184949.343543:0:13689:0:(sec.c:1727:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:1.0:1496184949.343544:0:13689:0:(genops.c:1030:class_import_put()) Process entered 00000020:00000040:1.0:1496184949.343545:0:13689:0:(genops.c:1037:class_import_put()) import ffff881043bd0000 refcount=2 obd=lustre-MDT0000-lwp-OST0000 00000020:00000001:1.0:1496184949.343546:0:13689:0:(genops.c:1046:class_import_put()) Process leaving 02000000:00000010:1.0:1496184949.343547:0:13689:0:(sec_null.c:188:null_free_reqbuf()) kfreed 'req->rq_reqbuf': 256 at ffff880822230300. 02000000:00000001:1.0:1496184949.343548:0:13689:0:(sec.c:466:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:1.0:1496184949.343549:0:13689:0:(sec.c:482:sptlrpc_req_put_ctx()) Process leaving 00000100:00000010:1.0:1496184949.343549:0:13689:0:(client.c:499:ptlrpc_request_cache_free()) slab-freed '(req)': 760 at ffff88080f769800. 00000100:00000001:1.0:1496184949.343551:0:13689:0:(client.c:2490:__ptlrpc_free_req()) Process leaving 00000100:00000001:1.0:1496184949.343552:0:13689:0:(client.c:2528:__ptlrpc_req_finished()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.343553:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184949.343555:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184949.343556:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343557:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184949.343559:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184949.343560:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184949.343561:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184949.366783:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.366785:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.366787:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0c50 x1568861104106368 msgsize 520 00000100:00100000:7.0:1496184949.366792:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.366793:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.366795:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.367148:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.367150:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.367152:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e689c00. 02000000:00000001:7.0:1496184949.367155:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.367156:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.367618:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.367619:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.367622:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.367627:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.367628:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.367630:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.369616:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.369618:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.369624:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.369628:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.369629:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.369634:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.397428:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.397430:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.397432:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd8c50 x1568861104106400 msgsize 520 00000100:00100000:6.0:1496184949.397437:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.397438:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.397440:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.397748:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.397750:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.397752:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68a000. 02000000:00000001:6.0:1496184949.397755:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.397756:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.398225:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.398227:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.398229:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026078450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.398234:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.398235:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.398237:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.400124:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.400127:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.411267:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.411272:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.411273:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.422266:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.433215:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.433217:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.433220:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c0050 x1568861104106432 msgsize 520 00000100:00100000:6.0:1496184949.433225:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.433226:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.433228:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.433565:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.433567:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.433568:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e688400. 02000000:00000001:7.0:1496184949.433571:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.433572:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.434011:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.434013:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.434023:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.434028:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.434029:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.434031:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.436007:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.436009:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.447049:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.447054:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.447055:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.458101:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184949.458195:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184949.458197:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184949.458201:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a850 x1568861104106464 msgsize 520 00000100:00100000:1.0:1496184949.458210:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.458211:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.458215:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184949.458221:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.458224:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.458228:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184949.458230:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184949.458232:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184949.458234:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184949.458238:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184949.458242:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106464 00000020:00000001:3.0:1496184949.458245:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184949.458246:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184949.458248:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184949.458251:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000020:00000001:3.0:1496184949.458253:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184949.458254:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184949.458259:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.458260:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184949.458262:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105bc400. 00000020:00000010:3.0:1496184949.458267:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed83080. 00000020:00000010:3.0:1496184949.458269:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184949.458275:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184949.458276:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184949.458277:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.458282:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184949.458288:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8a850 x1568861104106464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.458293:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184949.458295:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458299:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184949.458303:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.458304:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184949.458305:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.458306:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.458307:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184949.458309:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184949.458312:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6105 00000100:00000040:3.0:1496184949.458314:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.458316:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736390224 : -131906973161392 : ffff880803b8a850) 00000100:00000040:3.0:1496184949.458319:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a850 x1568861104106464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.458324:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.458325:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184949.458327:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106464:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184949.458329:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106464 00000020:00000001:3.0:1496184949.458331:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184949.458336:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184949.458343:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184949.458356:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184949.458357:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184949.458360:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23f9800. 02000000:00000001:3.0:1496184949.458362:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458364:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.458370:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184949.458372:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184949.458376:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184949.458378:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184949.458380:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184949.458383:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:3.0:1496184949.458387:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184949.458390:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184949.458392:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184949.458394:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184949.458396:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184949.458397:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184949.458399:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184949.458400:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184949.458403:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184949.458404:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184949.458406:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184949.458407:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184949.458409:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184949.458414:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184949.458416:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184949.458417:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184949.458420:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184949.458421:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184949.458423:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184949.458426:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184949.458427:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184949.458428:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184949.458430:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184949.458432:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184949.458433:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.458435:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184949.458437:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184949.458439:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184949.458441:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.458442:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.458443:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.458445:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184949.458447:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.458448:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184949.458449:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.458452:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.458454:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.458455:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458457:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184949.458460:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184949.458462:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184949.458465:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184949.458467:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.458468:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.458470:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458471:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.458472:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.458474:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.458476:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184949.458478:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184949.458480:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458482:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184949.458484:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184949.458485:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184949.458487:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184949.458488:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184949.458489:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184949.458491:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184949.458493:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.458495:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.458497:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184949.458499:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.458511:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184949.458513:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184949.458515:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184949.458516:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184949.458517:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184949.458519:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184949.458520:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.458522:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184949.458524:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184949.458526:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106464 00010000:00000001:3.0:1496184949.458528:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184949.458530:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8a850 x1568861104106464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184949.458537:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184949.458539:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184949.458540:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184949.458543:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184949.458545:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.458546:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.458548:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.458549:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184949.458551:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458552:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184949.458554:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184949.458558:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106464, offset 0 00000100:00000001:3.0:1496184949.458574:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.458575:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.458576:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.458577:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.458579:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184949.458580:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.458582:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184949.458584:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a850 x1568861104106464/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184949.458589:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106464:12345-10.37.248.124@o2ib1:8 Request procesed in 264us (387us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184949.458595:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6105 00000100:00000040:3.0:1496184949.458603:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184949.458605:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184949.458605:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184949.458609:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed83080. 00000020:00000010:3.0:1496184949.458611:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184949.458613:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105bc400. 00000020:00000040:3.0:1496184949.458614:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184949.458616:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184949.458619:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184949.458624:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.458625:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.458928:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.458931:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.458932:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23f9800. 02000000:00000001:7.0:1496184949.458935:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.458936:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.459343:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.459345:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.459348:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.459362:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.459363:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.459365:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.469921:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.469923:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.480916:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.480920:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.480921:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.491902:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184949.502847:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184949.502849:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184949.502852:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8a450 x1568861104106496 msgsize 520 00000100:00100000:1.0:1496184949.502858:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.502859:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.502863:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184949.502904:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.502907:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.502909:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184949.502911:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184949.502912:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184949.502914:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184949.502917:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184949.502920:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106496 00000020:00000001:3.0:1496184949.502922:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184949.502924:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184949.502926:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184949.502927:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:3.0:1496184949.502929:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184949.502933:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184949.502937:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.502938:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184949.502940:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff8810105bc400. 00000020:00000010:3.0:1496184949.502943:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100ed83a80. 00000020:00000010:3.0:1496184949.502946:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184949.502948:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184949.502949:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184949.502950:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.502954:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184949.502958:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8a450 x1568861104106496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.502964:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184949.502965:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.502970:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184949.502973:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.502974:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184949.502975:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.502976:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.502977:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184949.502979:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:3.0:1496184949.502981:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6106 00000100:00000040:3.0:1496184949.502983:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.502985:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736389200 : -131906973162416 : ffff880803b8a450) 00000100:00000040:3.0:1496184949.502987:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8a450 x1568861104106496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184949.502992:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.502993:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184949.502995:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106496:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184949.502997:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106496 00000020:00000001:3.0:1496184949.502999:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184949.503001:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184949.503006:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184949.503011:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184949.503012:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184949.503014:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fc000. 02000000:00000001:3.0:1496184949.503016:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503017:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.503021:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184949.503023:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184949.503025:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184949.503027:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184949.503029:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184949.503031:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:3.0:1496184949.503034:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184949.503035:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184949.503036:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184949.503038:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184949.503039:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184949.503040:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184949.503041:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184949.503042:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184949.503044:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184949.503045:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184949.503047:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184949.503048:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184949.503049:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184949.503053:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184949.503054:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184949.503056:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184949.503058:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184949.503059:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184949.503061:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184949.503062:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184949.503064:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184949.503065:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184949.503067:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184949.503069:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184949.503071:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.503072:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184949.503073:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184949.503075:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184949.503076:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.503077:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.503079:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.503081:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184949.503082:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.503083:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184949.503084:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.503086:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.503087:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.503089:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503090:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184949.503093:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184949.503095:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184949.503097:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184949.503100:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.503101:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.503102:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503103:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.503105:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.503106:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.503108:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184949.503109:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184949.503111:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503112:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184949.503114:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184949.503115:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184949.503117:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184949.503118:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184949.503119:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184949.503121:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184949.503122:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.503124:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.503125:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184949.503126:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.503128:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184949.503130:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184949.503132:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184949.503132:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184949.503134:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184949.503135:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184949.503136:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.503137:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184949.503138:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184949.503140:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106496 00010000:00000001:3.0:1496184949.503141:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184949.503144:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8a450 x1568861104106496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184949.503149:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184949.503151:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184949.503152:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184949.503155:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184949.503157:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.503158:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.503159:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.503161:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184949.503162:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503163:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184949.503165:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184949.503169:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106496, offset 0 00000100:00000001:3.0:1496184949.503187:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.503188:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.503190:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.503191:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.503192:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184949.503193:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.503194:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184949.503196:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8a450 x1568861104106496/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184949.503202:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106496:12345-10.37.248.124@o2ib1:8 Request procesed in 208us (348us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184949.503207:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6106 00000100:00000040:3.0:1496184949.503209:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184949.503210:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184949.503211:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184949.503213:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100ed83a80. 00000020:00000010:3.0:1496184949.503215:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184949.503217:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff8810105bc400. 00000020:00000040:3.0:1496184949.503219:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184949.503221:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184949.503223:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184949.503227:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.503228:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.503278:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.503280:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.503282:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fc000. 02000000:00000001:7.0:1496184949.503285:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.503286:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.503713:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.503715:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.503718:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.503723:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.503724:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.503726:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.505554:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.505556:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.505562:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.505566:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.505567:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184949.505571:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:7.0:1496184949.534473:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.534475:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.534477:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9450 x1568861104106528 msgsize 520 00000100:00100000:7.0:1496184949.534483:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.534484:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.534486:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:7.0:1496184949.534835:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.534837:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.534839:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68fc00. 02000000:00000001:7.0:1496184949.534842:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.534842:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.535254:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.535256:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.535258:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026075050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.535263:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.535265:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.535267:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.537100:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.537103:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.537121:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.537125:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.537126:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.537133:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.565544:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.565546:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.565549:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9050 x1568861104106560 msgsize 520 00000100:00100000:6.0:1496184949.565554:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.565555:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.565557:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.565903:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.565905:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.565907:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e689400. 02000000:00000001:6.0:1496184949.565910:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.565910:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.566320:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.566322:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.566325:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.566329:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.566331:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.566333:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.568080:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.568082:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.568088:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.568093:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.568093:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.568098:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:6.0:1496184949.595582:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.595584:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.595586:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9c50 x1568861104106592 msgsize 520 00000100:00100000:6.0:1496184949.595591:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.595592:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.595594:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:6.0:1496184949.595946:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.595948:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.595949:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68c800. 02000000:00000001:6.0:1496184949.595952:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.595953:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.596394:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.596396:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.596398:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026074450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.596403:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.596404:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.596406:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00000100:00000001:3.0:1496184949.597937:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.597939:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.609074:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.609079:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.609080:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:2.0:1496184949.613925:0:13688:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1496184949.613927:0:13688:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184949.613934:0:13688:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184949.613935:0:13688:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.613937:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.613940:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184949.613942:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184949.613943:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.620074:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000020:00000010:5.0:1496184949.620286:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.620288:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.620289:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.620291:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.620296:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.620298:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.630628:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.630629:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.630632:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bd9850 x1568861104106640 msgsize 520 00000100:00100000:7.0:1496184949.630637:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.630638:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.630640:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.630673:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.630675:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.630677:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.630678:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.630679:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.630680:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.630682:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.630684:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106640 00000020:00000001:5.0:1496184949.630686:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.630687:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.630688:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.630689:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.630691:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.630692:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.630694:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.630695:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.630697:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.630699:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88082ffe1a80. 00000020:00000010:5.0:1496184949.630700:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.630702:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.630703:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.630704:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.630706:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.630709:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bd9850 x1568861104106640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.630714:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.630715:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630720:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.630722:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.630723:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.630724:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.630725:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.630725:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.630726:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.630729:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43410 00000100:00000040:5.0:1496184949.630730:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.630732:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683654736 : -131872025896880 : ffff881026bd9850) 00000100:00000040:5.0:1496184949.630734:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bd9850 x1568861104106640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.630739:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.630740:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.630742:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106640:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.630744:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106640 00000020:00000001:5.0:1496184949.630745:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.630747:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.630750:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.630752:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.630753:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.630755:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68bc00. 02000000:00000001:5.0:1496184949.630756:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630757:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.630759:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.630761:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.630763:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.630764:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.630766:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.630768:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.630770:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.630771:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.630772:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.630773:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.630774:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.630775:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.630776:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.630778:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.630779:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.630780:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.630781:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.630782:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.630784:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.630787:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.630787:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.630789:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.630790:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.630791:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.630793:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.630795:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.630795:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.630796:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.630798:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.630799:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.630801:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.630802:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.630803:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.630804:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.630806:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.630807:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.630808:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.630809:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.630810:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.630811:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.630812:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.630814:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.630815:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.630817:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630818:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.630820:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.630822:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.630824:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.630825:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.630827:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.630828:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630829:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.630830:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.630831:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.630833:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.630834:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.630835:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630836:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.630837:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.630838:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.630839:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.630841:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.630841:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.630843:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.630844:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.630845:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.630847:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.630848:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.630849:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.630850:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.630852:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.630853:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.630854:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.630854:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.630855:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.630857:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.630858:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.630859:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106640 00010000:00000001:5.0:1496184949.630861:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.630863:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bd9850 x1568861104106640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.630868:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.630869:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.630870:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.630872:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.630874:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.630875:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.630876:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.630877:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.630878:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630879:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.630881:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.630883:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106640, offset 0 00000100:00000001:5.0:1496184949.630905:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.630907:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.630908:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.630910:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.630911:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.630911:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.630912:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.630914:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bd9850 x1568861104106640/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.630920:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106640:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (286us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.630924:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43410 00000100:00000040:5.0:1496184949.630926:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.630927:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.630928:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.630929:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88082ffe1a80. 00000020:00000010:5.0:1496184949.630931:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.630933:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.630934:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.630936:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.630938:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.630941:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.630942:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.631001:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.631004:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.631006:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68bc00. 02000000:00000001:6.0:1496184949.631011:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.631012:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.631449:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.631451:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.631453:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026074850 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.631458:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.631466:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.631468:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.631501:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.631504:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.631505:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.631506:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.631507:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.631508:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.631511:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.631512:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.631514:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.631515:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.631516:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.631517:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.631519:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.631520:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.631522:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.631523:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.631525:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.631527:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88082ffe1100. 00000020:00000010:5.0:1496184949.631528:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.631530:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.631532:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.631533:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.631534:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.631535:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.631536:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.631537:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.631538:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.631540:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.631541:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.631542:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.631543:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.631544:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.631545:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.631546:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.631546:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631547:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.631548:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631549:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631551:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.631552:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.631553:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.631554:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.631557:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.631563:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.631564:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.631566:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.631566:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.631567:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.631569:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.631570:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.631571:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.631572:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.631572:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00002000:00000001:5.0:1496184949.631574:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.631575:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.631577:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.631581:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.631584:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.631584:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.631585:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.631586:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.631587:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.631588:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.631591:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57678 00000100:00000040:5.0:1496184949.631592:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.631593:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671706704 : -131872037844912 : ffff881026074850) 00000100:00000040:5.0:1496184949.631596:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.631601:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.631601:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.631603:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.631605:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.631606:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.631608:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.631609:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631610:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.631611:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.631612:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.631613:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.631614:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.631615:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.631615:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631617:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.631618:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.631619:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.631621:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68f400. 02000000:00000001:5.0:1496184949.631622:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.631623:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631624:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.631625:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.631627:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.631631:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.631634:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.631635:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.631637:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.631639:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.631640:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.631641:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.631642:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.631644:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.631645:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.631647:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.631648:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.631649:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.631649:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.631650:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.631652:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.631653:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.631654:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.631656:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.631679:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.631684:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631686:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.631688:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.631690:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.631691:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.631692:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.631693:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.631694:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.631696:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.631697:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.631701:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.631702:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.631728:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfeca0-0x592def0cfeca0 00000100:00000001:5.0:1496184949.631730:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.633235:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.633237:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.644275:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.644280:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.644281:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.655321:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.655383:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.655402:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.655405:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000001:1.0:1496184949.655407:0:21159:0:(events.c:294:request_in_callback()) Process entered 00002000:00000001:5.0:1496184949.655409:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.655410:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000200:1.0:1496184949.655410:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:5.0:1496184949.655412:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00000100:00000040:1.0:1496184949.655413:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b850 x1568861104106672 msgsize 520 00002000:00000001:5.0:1496184949.655414:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184949.655419:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.655420:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.655424:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184949.655429:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.655432:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.655436:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184949.655439:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184949.655441:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184949.655442:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.655447:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184949.655447:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.655449:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000100:00100000:3.0:1496184949.655450:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106672 00002000:00000001:5.0:1496184949.655451:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.655452:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.655453:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1496184949.655453:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:5.0:1496184949.655454:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.655455:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000040:3.0:1496184949.655455:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.655456:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.655457:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000020:00000001:3.0:1496184949.655457:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:5.0:1496184949.655458:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.655459:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.655460:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000010:5.0:1496184949.655461:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000001:3.0:1496184949.655461:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000010:5.0:1496184949.655462:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000020:00000001:3.0:1496184949.655463:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.655464:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.655464:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.655466:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68f400. 02000000:00000001:5.0:1496184949.655467:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:3.0:1496184949.655467:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.655468:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:02000400:5.0:1496184949.655469:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.655471:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000010:3.0:1496184949.655472:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff880822299400. 00010000:00000080:5.0:1496184949.655473:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184949.655476:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100e924100. 00010000:00000040:5.0:1496184949.655479:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.655480:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.655481:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.655481:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:3.0:1496184949.655481:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000020:00000001:5.0:1496184949.655483:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000001:3.0:1496184949.655484:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000040:5.0:1496184949.655485:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184949.655485:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184949.655487:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.655492:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:5.0:1496184949.655495:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23892us (24040us total) trans 0 rc -110/-999 00000100:00000200:3.0:1496184949.655497:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8b850 x1568861104106672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:5.0:1496184949.655500:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57678 00000100:00000040:5.0:1496184949.655502:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184949.655503:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.655504:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184949.655504:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.655505:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.655507:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184949.655508:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184949.655512:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00002000:00000001:5.0:1496184949.655513:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00004000:00000001:3.0:1496184949.655513:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00002000:00100000:5.0:1496184949.655514:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074850 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184949.655514:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00010000:00000001:5.0:1496184949.655515:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00004000:00000001:3.0:1496184949.655515:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000020:00000001:5.0:1496184949.655516:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:3.0:1496184949.655516:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000020:00000001:5.0:1496184949.655518:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000100:00000001:3.0:1496184949.655518:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000001:5.0:1496184949.655519:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.655521:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.655521:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00000100:00100000:3.0:1496184949.655521:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6107 00002000:00000001:5.0:1496184949.655522:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.655523:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074850. 00000100:00000040:3.0:1496184949.655523:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00002000:00000001:5.0:1496184949.655524:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.655525:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:3.0:1496184949.655525:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736394320 : -131906973157296 : ffff880803b8b850) 00000020:00000010:5.0:1496184949.655527:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88082ffe1100. 00000100:00000040:3.0:1496184949.655528:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b850 x1568861104106672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000010:5.0:1496184949.655529:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.655530:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.655532:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:3.0:1496184949.655533:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.655534:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.655534:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00000200:5.0:1496184949.655536:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00100000:3.0:1496184949.655536:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106672:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184949.655539:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106672 00004000:00000001:5.0:1496184949.655541:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000020:00000001:3.0:1496184949.655541:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00004000:00000001:5.0:1496184949.655542:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00000001:3.0:1496184949.655545:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184949.655553:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184949.655558:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184949.655560:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184949.655562:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fc800. 02000000:00000001:3.0:1496184949.655564:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655566:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.655572:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184949.655582:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184949.655585:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184949.655587:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184949.655589:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184949.655591:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:3.0:1496184949.655595:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184949.655598:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184949.655601:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184949.655603:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184949.655605:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184949.655606:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184949.655608:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184949.655610:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184949.655612:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184949.655613:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184949.655615:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184949.655616:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184949.655618:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184949.655623:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184949.655624:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184949.655626:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184949.655629:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184949.655630:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184949.655632:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184949.655634:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184949.655635:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184949.655637:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184949.655639:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184949.655640:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184949.655642:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.655648:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184949.655650:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184949.655652:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184949.655654:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.655655:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.655657:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.655658:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184949.655661:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.655662:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184949.655663:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.655665:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.655667:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.655669:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655670:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184949.655674:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184949.655676:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184949.655679:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184949.655682:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.655683:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.655684:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655685:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.655687:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.655688:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.655690:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184949.655692:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184949.655694:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655696:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184949.655698:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184949.655699:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184949.655701:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184949.655703:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184949.655704:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184949.655705:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184949.655708:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.655710:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.655711:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184949.655713:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.655716:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184949.655718:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184949.655720:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184949.655721:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184949.655722:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184949.655723:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184949.655725:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.655727:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184949.655729:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184949.655731:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106672 00010000:00000001:3.0:1496184949.655733:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184949.655736:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8b850 x1568861104106672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184949.655748:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184949.655750:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184949.655751:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184949.655754:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184949.655756:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.655757:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.655758:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.655760:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184949.655761:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655763:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184949.655765:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184949.655769:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106672, offset 0 00000100:00000001:3.0:1496184949.655785:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.655786:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.655788:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.655789:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.655791:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184949.655792:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.655793:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184949.655795:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b850 x1568861104106672/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184949.655801:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106672:12345-10.37.248.124@o2ib1:8 Request procesed in 266us (387us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184949.655806:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6107 00000100:00000040:3.0:1496184949.655808:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184949.655809:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184949.655810:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184949.655813:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100e924100. 00000020:00000010:3.0:1496184949.655815:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184949.655817:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff880822299400. 00000020:00000040:3.0:1496184949.655826:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184949.655828:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184949.655831:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184949.655837:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.655838:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.656144:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.656146:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.656148:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fc800. 02000000:00000001:7.0:1496184949.656151:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.656152:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.656527:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.656529:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.656531:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026076850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.656536:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.656538:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.656540:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.656572:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.656575:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.656577:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.656578:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.656580:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.656581:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.656583:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.656585:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.656587:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.656588:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.656589:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.656590:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.656592:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.656594:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.656596:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.656597:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.656599:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.656602:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674180. 00000020:00000010:5.0:1496184949.656603:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.656606:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.656608:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.656609:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.656610:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.656611:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.656612:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.656613:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.656615:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.656616:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.656617:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.656618:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.656619:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.656620:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.656621:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.656621:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.656622:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656623:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.656625:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656626:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656627:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.656628:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.656629:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.656630:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.656633:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.656640:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.656640:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.656642:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.656643:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.656644:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.656645:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.656647:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.656647:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.656648:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.656649:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00002000:00000001:5.0:1496184949.656650:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.656651:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.656653:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.656658:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.656660:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.656661:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.656662:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.656663:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.656664:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.656665:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.656667:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57679 00000100:00000040:5.0:1496184949.656669:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.656670:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671714896 : -131872037836720 : ffff881026076850) 00000100:00000040:5.0:1496184949.656673:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.656678:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.656679:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.656680:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.656683:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.656684:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.656686:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.656687:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656688:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.656688:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.656690:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.656691:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.656692:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.656693:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.656694:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656695:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.656696:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.656697:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.656699:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68f400. 02000000:00000001:5.0:1496184949.656700:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.656701:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656703:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.656704:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.656706:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.656710:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.656713:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.656713:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.656716:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.656717:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.656719:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.656720:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.656721:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.656723:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.656724:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.656726:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.656728:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.656729:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.656729:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.656730:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.656732:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.656733:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.656734:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.656737:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.656758:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.656764:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656765:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.656768:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.656769:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.656770:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.656771:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.656772:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.656774:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.656775:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.656777:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.656781:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.656782:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.656809:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfecc0-0x592def0cfecc0 00000100:00000001:5.0:1496184949.656811:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.658230:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.658233:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.669232:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.669237:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.669238:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.680210:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.680283:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.680290:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.680292:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.680296:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.680297:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.680299:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.680301:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.680334:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.680336:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.680338:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.680339:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.680340:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.680342:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.680343:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.680344:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.680345:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.680346:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.680347:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.680349:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.680350:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184949.680352:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.680352:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.680354:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68f400. 02000000:00000001:5.0:1496184949.680355:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.680357:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.680358:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.680361:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.680366:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.680368:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.680368:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.680369:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.680370:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.680372:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.680378:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23699us (23845us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.680383:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57679 00000100:00000040:5.0:1496184949.680385:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.680386:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.680387:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.680390:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.680400:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.680401:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.680402:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.680403:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.680404:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.680406:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.680407:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.680408:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.680409:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.680409:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076850. 00002000:00000001:5.0:1496184949.680410:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.680412:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.680413:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674180. 00000020:00000010:5.0:1496184949.680415:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.680417:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.680418:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.680419:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.680422:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.680427:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.680428:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.691755:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.691756:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.691765:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bda850 x1568861104106704 msgsize 520 00000100:00100000:7.0:1496184949.691770:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.691771:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.691773:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.691817:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.691822:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.691824:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.691826:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.691828:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.691829:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.691833:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.691835:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106704 00000020:00000001:5.0:1496184949.691838:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.691839:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.691841:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.691842:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.691845:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.691847:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.691851:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.691852:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.691854:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.691857:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674600. 00000020:00000010:5.0:1496184949.691871:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.691872:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.691873:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.691874:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.691876:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.691879:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bda850 x1568861104106704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.691885:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.691885:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.691903:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.691905:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.691906:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.691907:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.691907:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.691908:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.691909:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.691911:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43411 00000100:00000040:5.0:1496184949.691913:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.691914:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683658832 : -131872025892784 : ffff881026bda850) 00000100:00000040:5.0:1496184949.691917:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda850 x1568861104106704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.691922:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.691922:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.691924:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106704:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.691926:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106704 00000020:00000001:5.0:1496184949.691927:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.691929:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.691933:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.691935:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.691936:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.691938:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68f400. 02000000:00000001:5.0:1496184949.691939:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.691940:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.691943:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.691945:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.691947:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.691948:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.691950:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.691952:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.691954:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.691956:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.691957:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.691958:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.691959:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.691960:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.691962:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.691963:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.691965:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.691965:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.691967:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.691968:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.691969:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.691972:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.691973:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.691974:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.691976:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.691977:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.691979:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.691981:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.691981:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.691982:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.691984:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.691986:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.691988:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.691989:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.691990:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.691992:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.691993:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.691994:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.691995:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.691997:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.691998:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.691999:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.692000:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.692002:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.692003:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.692004:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692006:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.692008:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.692010:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.692012:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.692014:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.692015:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.692016:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692017:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.692019:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.692020:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.692022:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.692023:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.692024:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692026:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.692027:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.692028:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.692029:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.692031:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.692031:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.692033:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.692035:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.692036:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.692037:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.692038:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692040:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.692041:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.692042:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.692043:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.692044:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.692045:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.692046:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692047:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.692048:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.692050:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106704 00010000:00000001:5.0:1496184949.692051:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.692053:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bda850 x1568861104106704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.692058:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.692059:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.692061:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.692063:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.692065:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.692066:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.692067:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.692069:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.692069:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692071:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.692072:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.692074:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106704, offset 0 00000100:00000001:5.0:1496184949.692085:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692086:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.692088:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.692089:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.692090:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.692091:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692092:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.692093:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda850 x1568861104106704/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.692099:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106704:12345-10.37.248.124@o2ib1:8 Request procesed in 176us (339us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.692103:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43411 00000100:00000040:5.0:1496184949.692105:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.692106:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.692107:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.692108:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674600. 00000020:00000010:5.0:1496184949.692110:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.692111:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.692113:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.692114:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.692116:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.692120:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.692121:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.692185:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.692188:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.692191:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68f400. 02000000:00000001:6.0:1496184949.692195:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.692196:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.692669:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.692671:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.692673:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075850 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.692678:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.692679:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.692681:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.692714:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.692717:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.692718:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.692719:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.692720:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.692721:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.692724:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.692726:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.692727:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.692728:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.692729:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.692731:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.692732:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.692734:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.692736:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.692737:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.692739:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.692741:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674d00. 00000020:00000010:5.0:1496184949.692742:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.692744:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.692746:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.692746:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.692747:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.692749:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.692750:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.692751:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.692752:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.692754:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.692754:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.692756:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.692756:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.692758:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.692758:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.692759:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.692760:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692761:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.692762:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692763:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692764:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.692766:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.692767:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.692768:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.692771:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.692777:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.692778:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.692780:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.692780:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.692781:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.692783:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.692784:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.692785:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.692786:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.692786:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00002000:00000001:5.0:1496184949.692787:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692789:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692791:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692795:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.692797:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.692798:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.692799:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.692800:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.692801:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.692802:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.692804:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57680 00000100:00000040:5.0:1496184949.692806:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.692807:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671710800 : -131872037840816 : ffff881026075850) 00000100:00000040:5.0:1496184949.692809:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.692814:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.692815:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.692817:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.692819:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.692820:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.692822:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.692822:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692824:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.692824:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.692826:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.692827:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.692828:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.692829:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.692829:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692830:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.692832:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.692833:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.692835:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e689800. 02000000:00000001:5.0:1496184949.692836:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692837:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692838:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.692839:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.692841:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.692845:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.692848:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.692849:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.692851:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.692853:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.692854:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.692855:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.692856:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.692858:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.692859:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.692861:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.692862:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.692863:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.692863:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.692864:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.692866:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.692867:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.692868:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.692870:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.692904:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.692910:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692912:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.692914:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.692916:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.692917:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.692918:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.692919:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.692920:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.692922:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.692923:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.692927:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.692928:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.692955:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfece0-0x592def0cfece0 00000100:00000001:5.0:1496184949.692958:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.694662:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.694665:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.705704:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.705709:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.705710:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.716685:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.716747:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.716766:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.716768:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.716772:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.716773:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.716776:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.716777:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.716810:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.716813:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.716814:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.716815:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.716816:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.716817:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.716819:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.716820:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.716821:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.716821:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.716822:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.716824:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.716826:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184949.716827:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.716828:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.716829:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e689800. 02000000:00000001:5.0:1496184949.716831:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.716832:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.716834:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.716837:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.716842:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.716843:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.716844:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.716844:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.716846:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.716847:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.716853:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24038us (24179us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.716858:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57680 00000100:00000040:5.0:1496184949.716860:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.716861:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.716862:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.716865:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.716870:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.716871:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.716872:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.716873:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.716875:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.716876:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.716878:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.716878:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.716879:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.716880:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075850. 00002000:00000001:5.0:1496184949.716881:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.716882:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.716884:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674d00. 00000020:00000010:5.0:1496184949.716885:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.716887:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.716888:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.716900:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.716903:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.716908:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.716909:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.727616:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.727618:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.727620:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bda450 x1568861104106736 msgsize 520 00000100:00100000:7.0:1496184949.727625:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.727626:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.727628:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.727660:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.727662:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.727664:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.727665:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.727666:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.727667:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.727669:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.727671:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106736 00000020:00000001:5.0:1496184949.727673:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.727673:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.727675:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.727676:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.727677:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.727679:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.727681:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.727682:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.727683:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.727685:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674c00. 00000020:00000010:5.0:1496184949.727687:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.727688:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.727689:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.727690:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.727692:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.727696:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bda450 x1568861104106736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.727701:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.727702:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727707:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.727709:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.727710:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.727710:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.727711:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.727712:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.727713:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.727715:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43412 00000100:00000040:5.0:1496184949.727717:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.727718:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683657808 : -131872025893808 : ffff881026bda450) 00000100:00000040:5.0:1496184949.727720:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bda450 x1568861104106736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.727725:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.727726:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.727728:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106736:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.727730:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106736 00000020:00000001:5.0:1496184949.727731:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.727733:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.727736:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.727739:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.727740:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.727741:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e689800. 02000000:00000001:5.0:1496184949.727742:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727743:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.727746:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.727747:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.727749:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.727750:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.727752:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.727754:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.727757:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.727758:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.727759:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.727760:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.727761:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.727762:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.727763:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.727764:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.727766:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.727767:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.727768:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.727769:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.727770:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.727773:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.727774:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.727775:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.727777:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.727778:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.727780:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.727781:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.727782:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.727783:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.727785:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.727786:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.727788:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.727789:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.727790:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.727791:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.727793:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.727794:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.727795:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.727796:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.727798:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.727799:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.727799:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.727801:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.727802:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.727803:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727804:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.727807:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.727809:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.727811:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.727812:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.727813:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.727814:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727816:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.727817:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.727818:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.727820:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.727821:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.727822:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727823:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.727825:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.727825:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.727826:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.727828:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.727829:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.727830:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.727831:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.727833:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.727834:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.727835:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.727837:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.727838:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.727839:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.727840:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.727841:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.727842:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.727843:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.727844:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.727845:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.727846:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106736 00010000:00000001:5.0:1496184949.727848:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.727849:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bda450 x1568861104106736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.727855:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.727855:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.727857:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.727859:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.727860:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.727862:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.727863:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.727864:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.727865:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727866:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.727868:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.727870:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106736, offset 0 00000100:00000001:5.0:1496184949.727880:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.727881:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.727883:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.727884:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.727885:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.727886:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.727887:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.727888:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bda450 x1568861104106736/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.727910:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106736:12345-10.37.248.124@o2ib1:8 Request procesed in 183us (288us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.727915:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43412 00000100:00000040:5.0:1496184949.727916:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.727917:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.727918:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.727920:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674c00. 00000020:00000010:5.0:1496184949.727921:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.727923:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.727924:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.727926:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.727927:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.727931:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.727931:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.727963:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.727966:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.727967:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e689800. 02000000:00000001:7.0:1496184949.727970:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.727971:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.728436:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.728438:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.728440:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.728445:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.728446:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.728456:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.728485:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.728488:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.728490:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.728491:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.728492:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.728493:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.728495:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.728497:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.728498:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.728499:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.728500:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.728502:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.728503:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.728505:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.728507:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.728507:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.728509:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.728511:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674b00. 00000020:00000010:5.0:1496184949.728513:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.728515:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.728516:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.728517:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.728518:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.728519:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.728520:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.728522:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.728523:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.728524:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.728525:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.728526:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.728527:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.728528:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.728529:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.728530:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.728530:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728531:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.728533:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728534:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728535:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.728536:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.728537:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.728538:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.728541:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.728548:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.728548:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.728550:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.728551:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.728552:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.728553:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.728555:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.728555:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.728556:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.728556:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00002000:00000001:5.0:1496184949.728558:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.728559:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.728561:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.728566:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.728568:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.728569:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.728570:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.728571:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.728572:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.728572:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.728575:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57681 00000100:00000040:5.0:1496184949.728577:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.728578:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671703632 : -131872037847984 : ffff881026073c50) 00000100:00000040:5.0:1496184949.728580:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.728585:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.728586:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.728588:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.728590:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.728591:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.728593:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.728593:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728594:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.728595:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.728597:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.728598:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.728599:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.728599:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.728600:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728601:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.728603:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.728604:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.728605:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e688000. 02000000:00000001:5.0:1496184949.728606:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.728608:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728609:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.728610:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.728612:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.728616:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.728619:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.728619:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.728622:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.728624:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.728625:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.728625:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.728627:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.728629:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.728630:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.728632:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.728633:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.728634:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.728634:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.728635:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.728637:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.728638:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.728639:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.728641:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.728663:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.728668:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728670:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.728672:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.728674:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.728675:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.728676:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.728677:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.728678:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.728680:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.728681:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.728686:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.728687:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.728713:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfed00-0x592def0cfed00 00000100:00000001:5.0:1496184949.728715:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.730252:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.730255:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.741358:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.741363:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.741367:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.752366:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.752429:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.752448:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.752450:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.752454:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.752455:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.752457:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.752459:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.752492:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.752494:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.752496:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.752497:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.752498:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.752499:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.752500:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.752502:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.752503:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.752503:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.752504:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.752506:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.752507:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184949.752509:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.752510:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.752511:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e688000. 02000000:00000001:5.0:1496184949.752512:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.752514:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.752516:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.752518:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.752524:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.752525:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.752526:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.752527:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.752528:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.752530:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.752536:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23949us (24094us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.752540:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57681 00000100:00000040:5.0:1496184949.752542:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.752543:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.752544:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.752547:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.752552:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.752553:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.752555:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.752555:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.752557:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.752558:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.752560:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.752560:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.752561:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.752562:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073c50. 00002000:00000001:5.0:1496184949.752563:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.752564:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.752566:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674b00. 00000020:00000010:5.0:1496184949.752567:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.752569:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.752570:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.752572:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.752574:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.752580:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.752581:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.763243:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.763245:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.763247:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdac50 x1568861104106768 msgsize 520 00000100:00100000:6.0:1496184949.763252:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.763253:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.763256:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.763288:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.763290:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.763292:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.763293:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.763294:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.763295:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.763297:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.763299:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106768 00000020:00000001:5.0:1496184949.763301:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.763301:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.763303:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.763304:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.763305:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.763307:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.763309:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.763310:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.763311:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.763313:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674e80. 00000020:00000010:5.0:1496184949.763315:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.763316:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.763317:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.763318:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.763320:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.763323:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdac50 x1568861104106768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.763329:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.763329:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763334:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.763336:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.763337:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.763338:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.763339:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.763340:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.763341:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.763343:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43413 00000100:00000040:5.0:1496184949.763345:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.763346:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683659856 : -131872025891760 : ffff881026bdac50) 00000100:00000040:5.0:1496184949.763348:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdac50 x1568861104106768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.763354:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.763354:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.763356:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106768:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.763358:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106768 00000020:00000001:5.0:1496184949.763359:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.763361:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.763364:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.763367:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.763368:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.763369:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e688000. 02000000:00000001:5.0:1496184949.763370:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763371:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.763374:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.763375:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.763377:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.763379:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.763380:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.763382:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.763385:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.763386:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.763387:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.763388:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.763389:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.763390:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.763391:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.763392:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.763394:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.763394:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.763396:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.763397:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.763398:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.763401:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.763402:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.763403:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.763405:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.763406:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.763408:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.763409:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.763410:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.763411:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.763413:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.763414:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.763416:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.763417:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.763418:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.763419:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.763421:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.763422:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.763423:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.763424:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.763425:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.763426:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.763427:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.763429:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.763430:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.763432:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763433:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.763435:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.763438:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.763439:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.763441:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.763442:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.763443:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763444:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.763445:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.763446:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.763448:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.763449:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.763450:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763451:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.763452:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.763453:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.763454:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.763456:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.763456:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.763458:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.763459:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.763460:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.763462:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.763463:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.763464:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.763465:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.763467:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.763468:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.763469:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.763470:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.763471:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.763472:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.763473:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.763474:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106768 00010000:00000001:5.0:1496184949.763476:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.763478:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdac50 x1568861104106768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.763483:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.763484:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.763485:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.763487:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.763489:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.763490:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.763491:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.763492:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.763493:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763494:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.763496:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.763498:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106768, offset 0 00000100:00000001:5.0:1496184949.763508:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.763509:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.763511:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.763512:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.763513:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.763514:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.763515:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.763516:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdac50 x1568861104106768/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.763522:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106768:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (273us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.763526:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43413 00000100:00000040:5.0:1496184949.763528:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.763529:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.763530:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.763531:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674e80. 00000020:00000010:5.0:1496184949.763533:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.763534:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.763536:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.763537:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.763539:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.763542:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.763543:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.763607:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.763609:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.763611:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e688000. 02000000:00000001:7.0:1496184949.763614:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.763615:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.764130:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.764132:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.764135:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026077450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.764140:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.764141:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.764143:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.764173:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.764175:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.764177:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.764178:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.764179:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.764180:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.764183:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.764184:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.764186:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.764187:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.764188:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.764189:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.764191:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.764192:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.764194:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.764195:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.764197:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.764199:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674580. 00000020:00000010:5.0:1496184949.764200:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.764202:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.764204:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.764205:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.764206:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.764207:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.764208:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.764209:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.764210:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.764211:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.764213:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.764214:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.764215:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.764216:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.764217:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.764218:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.764218:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764219:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.764221:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764222:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764223:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.764224:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.764225:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.764226:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.764229:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.764235:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.764236:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.764238:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.764238:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.764239:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.764241:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.764242:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.764243:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.764244:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.764244:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00002000:00000001:5.0:1496184949.764246:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.764247:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.764249:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.764253:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.764255:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.764256:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.764257:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.764258:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.764259:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.764260:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.764263:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57682 00000100:00000040:5.0:1496184949.764264:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.764265:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671717968 : -131872037833648 : ffff881026077450) 00000100:00000040:5.0:1496184949.764267:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.764273:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.764273:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.764275:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.764277:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.764278:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.764280:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.764281:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764282:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.764282:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.764284:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.764285:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.764286:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.764287:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.764287:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764288:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.764290:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.764291:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.764292:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e688c00. 02000000:00000001:5.0:1496184949.764294:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.764295:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764296:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.764297:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.764299:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.764303:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.764305:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.764306:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.764308:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.764310:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.764311:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.764312:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.764314:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.764316:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.764316:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.764318:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.764320:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.764321:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.764321:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.764322:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.764324:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.764324:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.764325:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.764328:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.764350:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.764356:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764357:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.764360:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.764361:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.764362:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.764363:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.764364:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.764365:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.764367:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.764368:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.764373:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.764374:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.764400:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfed20-0x592def0cfed20 00000100:00000001:5.0:1496184949.764402:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.765835:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.765838:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.776873:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.776878:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.776879:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.787871:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.787933:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.787951:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.787954:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.787958:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.787959:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.787961:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.787963:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.787996:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.787998:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.788000:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.788001:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.788001:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.788003:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.788004:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.788005:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.788006:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.788007:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.788008:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.788009:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.788011:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184949.788012:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.788013:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.788015:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e688c00. 02000000:00000001:5.0:1496184949.788016:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.788018:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.788019:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.788022:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.788027:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.788028:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.788029:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.788030:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.788031:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.788033:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.788043:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23768us (23906us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.788047:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57682 00000100:00000040:5.0:1496184949.788049:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.788050:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.788051:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.788054:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026077450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.788060:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.788060:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026077450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.788062:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.788063:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.788064:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.788066:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.788067:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.788068:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.788068:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.788069:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026077450. 00002000:00000001:5.0:1496184949.788070:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.788071:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.788073:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674580. 00000020:00000010:5.0:1496184949.788074:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.788076:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.788077:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.788079:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.788081:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.788086:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.788087:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.798932:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.798934:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.798937:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb050 x1568861104106800 msgsize 520 00000100:00100000:6.0:1496184949.798942:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.798943:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.798945:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.798979:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.798982:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.798983:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.798984:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.798986:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.798987:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.798989:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.798991:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106800 00000020:00000001:5.0:1496184949.798992:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.798993:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.798994:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.798995:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.798997:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.798999:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.799001:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.799002:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.799003:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.799005:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674f80. 00000020:00000010:5.0:1496184949.799007:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.799008:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.799009:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.799010:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.799012:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.799015:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdb050 x1568861104106800/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.799020:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.799021:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799026:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.799028:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.799029:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.799029:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.799030:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.799031:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.799032:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.799034:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43414 00000100:00000040:5.0:1496184949.799036:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.799037:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683660880 : -131872025890736 : ffff881026bdb050) 00000100:00000040:5.0:1496184949.799040:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb050 x1568861104106800/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.799045:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.799045:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.799047:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106800:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.799049:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106800 00000020:00000001:5.0:1496184949.799050:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.799052:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.799055:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.799058:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.799059:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.799060:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e688c00. 02000000:00000001:5.0:1496184949.799062:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799063:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.799065:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.799067:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.799069:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.799070:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.799071:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.799073:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.799076:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.799077:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.799078:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.799079:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.799080:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.799081:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.799082:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.799083:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.799085:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.799086:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.799087:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.799088:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.799089:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.799092:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.799093:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.799094:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.799096:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.799097:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.799099:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.799100:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.799101:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.799102:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.799104:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.799105:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.799107:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799108:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.799109:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.799110:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.799112:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799113:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.799114:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.799115:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.799116:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.799117:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.799118:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.799120:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.799121:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.799123:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799124:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.799126:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.799128:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.799130:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.799131:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.799133:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.799134:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799135:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.799136:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.799137:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.799139:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.799140:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.799141:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799142:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.799144:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.799144:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.799145:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.799147:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.799148:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.799149:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.799150:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.799152:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.799153:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.799154:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799156:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.799157:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.799158:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.799159:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.799160:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.799161:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.799162:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799163:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.799164:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.799165:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106800 00010000:00000001:5.0:1496184949.799167:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.799168:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdb050 x1568861104106800/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.799174:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.799175:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.799176:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.799178:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.799180:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.799181:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.799183:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.799184:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.799185:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799186:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.799187:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.799189:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106800, offset 0 00000100:00000001:5.0:1496184949.799200:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799201:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.799202:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.799203:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.799204:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.799205:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799206:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.799208:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb050 x1568861104106800/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.799213:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106800:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (275us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.799218:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43414 00000100:00000040:5.0:1496184949.799220:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.799221:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.799222:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.799223:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674f80. 00000020:00000010:5.0:1496184949.799225:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.799226:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.799228:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.799229:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.799231:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.799234:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.799235:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.799307:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.799310:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.799312:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e688c00. 02000000:00000001:6.0:1496184949.799317:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.799318:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.799742:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.799744:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.799746:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026075c50 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.799751:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.799757:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.799759:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.799791:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.799794:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.799795:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.799796:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.799797:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.799798:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.799801:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.799803:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.799804:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.799805:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.799806:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.799807:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.799809:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.799811:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.799813:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.799813:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.799815:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.799817:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674800. 00000020:00000010:5.0:1496184949.799819:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.799822:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.799823:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.799824:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.799825:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.799826:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.799827:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.799828:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.799829:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.799831:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.799832:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.799833:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.799834:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.799835:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.799835:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.799836:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.799837:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799838:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.799839:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799841:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799842:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799843:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.799844:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799845:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.799848:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.799854:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.799855:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.799857:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.799858:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.799858:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.799860:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.799861:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.799862:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.799863:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.799863:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00002000:00000001:5.0:1496184949.799865:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799866:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799868:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799872:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.799875:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.799875:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.799876:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.799877:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.799878:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.799879:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.799881:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57683 00000100:00000040:5.0:1496184949.799883:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.799884:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671711824 : -131872037839792 : ffff881026075c50) 00000100:00000040:5.0:1496184949.799886:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.799905:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.799906:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.799908:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.799910:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.799912:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.799913:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.799914:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799915:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.799916:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.799917:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.799918:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.799920:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.799920:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.799921:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799922:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.799924:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.799925:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.799926:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68c000. 02000000:00000001:5.0:1496184949.799928:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799929:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799930:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.799931:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.799933:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.799937:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.799940:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.799941:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.799943:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.799945:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.799946:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.799946:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.799948:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.799950:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.799951:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.799953:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.799954:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.799955:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.799956:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.799956:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.799958:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.799959:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.799960:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.799962:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.799984:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.799989:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799991:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.799994:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799995:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.799996:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.799997:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.799998:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.799999:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.800001:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.800002:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.800007:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.800008:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.800034:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfed40-0x592def0cfed40 00000100:00000001:5.0:1496184949.800036:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.801619:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.801622:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.812643:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.812648:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.812648:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.823633:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.823696:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.823715:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.823718:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.823721:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.823722:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.823725:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.823726:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.823759:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.823762:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.823764:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.823765:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.823766:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.823767:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.823768:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.823770:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.823770:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.823771:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.823772:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.823774:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.823775:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184949.823777:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.823778:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.823779:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68c000. 02000000:00000001:5.0:1496184949.823780:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.823782:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.823784:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.823786:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.823791:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.823793:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.823793:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.823794:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.823795:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.823797:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.823803:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23896us (24055us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.823808:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57683 00000100:00000040:5.0:1496184949.823809:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.823811:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.823812:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.823815:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026075c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.823820:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.823821:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026075c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.823822:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.823823:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.823824:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.823826:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.823827:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.823828:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.823829:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.823829:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026075c50. 00002000:00000001:5.0:1496184949.823831:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.823832:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.823833:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674800. 00000020:00000010:5.0:1496184949.823835:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.823836:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.823838:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.823839:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.823841:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.823847:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.823848:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.834390:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.834392:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.834394:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdbc50 x1568861104106832 msgsize 520 00000100:00100000:7.0:1496184949.834399:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.834400:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.834402:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.834436:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.834439:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.834440:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.834441:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.834442:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.834443:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.834446:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.834448:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106832 00000020:00000001:5.0:1496184949.834449:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.834450:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.834451:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.834452:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.834454:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.834455:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.834458:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.834459:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.834460:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.834462:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674100. 00000020:00000010:5.0:1496184949.834464:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.834466:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.834466:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.834467:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.834469:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.834473:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdbc50 x1568861104106832/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.834477:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.834478:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834483:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.834485:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.834486:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.834487:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.834487:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.834488:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.834489:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.834492:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43415 00000100:00000040:5.0:1496184949.834493:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.834494:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683663952 : -131872025887664 : ffff881026bdbc50) 00000100:00000040:5.0:1496184949.834497:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdbc50 x1568861104106832/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.834502:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.834503:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.834504:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106832:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.834507:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106832 00000020:00000001:5.0:1496184949.834508:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.834510:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.834513:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.834515:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.834516:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.834518:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68c000. 02000000:00000001:5.0:1496184949.834519:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834520:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.834523:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.834524:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.834526:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.834528:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.834529:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.834531:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.834534:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.834535:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.834536:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.834537:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.834538:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.834538:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.834539:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.834541:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.834542:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.834543:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.834544:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.834545:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.834547:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.834550:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.834550:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.834552:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.834554:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.834554:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.834556:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.834558:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.834559:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.834560:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.834562:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.834563:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.834565:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.834566:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.834566:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.834568:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.834569:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.834570:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.834571:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.834573:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.834574:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.834575:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.834576:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.834577:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.834579:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.834580:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834581:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.834584:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.834586:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.834587:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.834589:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.834590:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.834591:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834592:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.834594:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.834595:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.834596:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.834598:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.834599:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834600:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.834601:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.834602:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.834603:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.834605:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.834605:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.834607:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.834608:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.834610:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.834611:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.834612:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.834613:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.834614:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.834616:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.834617:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.834618:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.834618:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.834619:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.834620:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.834622:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.834623:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106832 00010000:00000001:5.0:1496184949.834624:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.834626:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdbc50 x1568861104106832/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.834632:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.834633:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.834634:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.834636:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.834638:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.834639:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.834640:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.834641:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.834642:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834643:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.834645:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.834647:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106832, offset 0 00000100:00000001:5.0:1496184949.834658:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.834659:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.834660:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.834661:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.834662:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.834663:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.834664:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.834666:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdbc50 x1568861104106832/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.834671:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106832:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (276us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.834676:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43415 00000100:00000040:5.0:1496184949.834677:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.834678:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.834679:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.834681:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674100. 00000020:00000010:5.0:1496184949.834682:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.834684:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.834685:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.834687:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.834688:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.834692:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.834692:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.834748:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.834750:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.834752:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68c000. 02000000:00000001:6.0:1496184949.834754:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.834755:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.835203:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.835205:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.835207:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026076450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.835212:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.835213:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.835219:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.835252:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.835254:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.835256:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.835257:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.835258:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.835259:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.835262:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.835264:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.835265:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.835266:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.835267:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.835268:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.835270:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.835272:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.835274:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.835275:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.835277:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.835279:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674980. 00000020:00000010:5.0:1496184949.835280:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.835282:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.835283:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.835284:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.835285:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.835286:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.835287:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.835289:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.835290:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.835291:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.835292:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.835294:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.835294:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.835296:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.835296:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.835297:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.835298:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835299:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.835300:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835301:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835302:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.835303:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.835304:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.835306:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.835308:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.835315:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.835316:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.835317:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.835318:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.835319:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.835321:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.835322:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.835323:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.835323:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.835324:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00002000:00000001:5.0:1496184949.835325:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.835326:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.835328:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.835333:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.835335:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.835336:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.835337:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.835338:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.835339:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.835339:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.835342:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57684 00000100:00000040:5.0:1496184949.835343:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.835345:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671713872 : -131872037837744 : ffff881026076450) 00000100:00000040:5.0:1496184949.835347:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.835352:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.835353:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.835355:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.835357:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.835358:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.835360:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.835360:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835361:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.835362:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.835364:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.835365:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.835366:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.835367:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.835367:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835368:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.835370:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.835371:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.835372:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e689000. 02000000:00000001:5.0:1496184949.835374:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.835375:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835376:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.835377:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.835379:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.835383:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.835386:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.835387:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.835389:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.835391:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.835392:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.835393:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.835394:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.835396:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.835397:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.835399:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.835400:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.835401:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.835402:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.835403:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.835404:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.835405:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.835406:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.835409:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.835429:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.835435:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835436:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.835439:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.835440:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.835441:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.835442:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.835443:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.835445:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.835446:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.835447:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.835452:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.835453:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.835480:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfed60-0x592def0cfed60 00000100:00000001:5.0:1496184949.835482:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.837097:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.837100:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.848143:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.848148:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.848149:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.859154:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.859216:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.859236:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.859238:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000001:1.0:1496184949.859238:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184949.859241:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:5.0:1496184949.859242:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.859243:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000040:1.0:1496184949.859244:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8b050 x1568861104106864 msgsize 520 00002000:00000001:5.0:1496184949.859245:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.859247:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184949.859250:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184949.859252:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184949.859255:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184949.859261:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184949.859264:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.859268:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184949.859270:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184949.859272:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184949.859274:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184949.859278:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.859280:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.859282:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000100:00100000:3.0:1496184949.859282:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106864 00002000:00000001:5.0:1496184949.859284:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.859285:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00000020:00000001:3.0:1496184949.859285:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:5.0:1496184949.859286:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.859287:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000040:3.0:1496184949.859287:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.859288:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1496184949.859289:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.859290:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.859290:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.859291:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:3.0:1496184949.859291:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000040:5.0:1496184949.859292:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000020:00000001:3.0:1496184949.859293:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000010:5.0:1496184949.859294:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000001:3.0:1496184949.859295:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000010:5.0:1496184949.859296:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184949.859297:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.859298:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.859299:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e689000. 00000100:00000001:3.0:1496184949.859300:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184949.859300:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 02000000:00000001:5.0:1496184949.859301:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.859303:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000010:3.0:1496184949.859303:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100f3e7000. 00000020:00000001:5.0:1496184949.859304:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.859307:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184949.859310:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8080. 00010000:00000040:5.0:1496184949.859312:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000010:3.0:1496184949.859312:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00010000:00000001:5.0:1496184949.859314:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.859314:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.859315:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859315:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184949.859316:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.859317:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000001:3.0:1496184949.859317:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:5.0:1496184949.859318:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184949.859322:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:5.0:1496184949.859324:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23971us (24115us total) trans 0 rc -110/-999 00000100:00000200:3.0:1496184949.859328:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8b050 x1568861104106864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:5.0:1496184949.859329:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57684 00000100:00000040:5.0:1496184949.859331:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.859333:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.859334:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000001:3.0:1496184949.859334:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184949.859335:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.859336:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026076450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184949.859339:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00002000:00000001:5.0:1496184949.859342:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.859342:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026076450 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184949.859343:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00010000:00000001:5.0:1496184949.859344:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00004000:00000001:3.0:1496184949.859344:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:00000001:5.0:1496184949.859345:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00004000:00000001:3.0:1496184949.859345:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00000020:00000001:5.0:1496184949.859346:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00004000:00000001:3.0:1496184949.859346:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184949.859347:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00010000:00000001:5.0:1496184949.859348:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.859349:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00000100:00000001:3.0:1496184949.859349:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000001:5.0:1496184949.859350:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.859351:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.859352:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026076450. 00000100:00100000:3.0:1496184949.859352:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6108 00002000:00000001:5.0:1496184949.859353:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.859354:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.859355:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.859356:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674980. 00000100:00000001:3.0:1496184949.859356:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736392272 : -131906973159344 : ffff880803b8b050) 00000020:00000010:5.0:1496184949.859358:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.859359:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000100:00000040:3.0:1496184949.859359:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8b050 x1568861104106864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000040:5.0:1496184949.859361:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:5.0:1496184949.859363:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184949.859364:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000200:5.0:1496184949.859365:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00000001:3.0:1496184949.859365:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184949.859367:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106864:12345-10.37.248.124@o2ib1:8 00004000:00000001:5.0:1496184949.859370:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000100:00000200:3.0:1496184949.859370:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106864 00004000:00000001:5.0:1496184949.859371:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:00000001:3.0:1496184949.859372:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184949.859376:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184949.859383:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184949.859388:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184949.859390:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184949.859393:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23ff800. 02000000:00000001:3.0:1496184949.859395:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859396:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.859403:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184949.859406:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184949.859408:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184949.859410:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184949.859412:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184949.859422:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:3.0:1496184949.859426:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184949.859429:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184949.859432:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184949.859434:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184949.859436:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184949.859437:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184949.859439:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184949.859441:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184949.859443:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184949.859444:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184949.859446:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184949.859447:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184949.859449:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184949.859454:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184949.859455:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184949.859457:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184949.859459:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184949.859461:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184949.859463:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184949.859465:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184949.859466:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184949.859468:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184949.859470:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184949.859471:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184949.859474:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.859475:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184949.859477:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184949.859479:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184949.859481:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184949.859482:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.859483:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184949.859485:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184949.859487:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.859488:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184949.859489:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184949.859492:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.859494:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.859495:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859497:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184949.859500:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184949.859502:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184949.859505:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184949.859507:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.859509:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.859510:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859511:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.859512:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.859514:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.859516:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184949.859518:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184949.859520:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859521:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184949.859524:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184949.859525:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184949.859526:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184949.859528:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184949.859530:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184949.859532:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184949.859534:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184949.859535:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184949.859537:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184949.859539:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.859542:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184949.859544:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184949.859546:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184949.859547:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184949.859548:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184949.859549:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184949.859560:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.859562:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184949.859564:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184949.859566:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106864 00010000:00000001:3.0:1496184949.859568:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184949.859570:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8b050 x1568861104106864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184949.859577:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184949.859578:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184949.859580:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=103 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184949.859583:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184949.859585:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184949.859586:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.859588:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184949.859589:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184949.859591:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859592:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184949.859594:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184949.859598:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106864, offset 0 00000100:00000001:3.0:1496184949.859615:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.859616:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184949.859618:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184949.859620:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184949.859621:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184949.859622:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184949.859623:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184949.859625:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8b050 x1568861104106864/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184949.859631:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106864:12345-10.37.248.124@o2ib1:8 Request procesed in 266us (385us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184949.859636:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6108 00000100:00000040:3.0:1496184949.859638:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184949.859640:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184949.859641:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184949.859644:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8080. 00000020:00000010:3.0:1496184949.859654:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184949.859656:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100f3e7000. 00000020:00000040:3.0:1496184949.859657:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184949.859659:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184949.859662:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184949.859668:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184949.859669:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.859970:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.859973:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.859975:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23ff800. 02000000:00000001:7.0:1496184949.859985:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.859986:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.860382:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.860384:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.860386:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.860392:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.860393:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.860395:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.860426:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.860429:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.860431:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.860432:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.860433:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.860434:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.860446:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.860448:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.860449:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.860450:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.860451:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.860453:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.860454:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.860456:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.860458:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.860459:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.860461:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.860464:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88103e674c80. 00000020:00000010:5.0:1496184949.860465:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.860468:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.860469:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.860470:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.860471:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.860473:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.860474:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.860475:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.860476:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.860478:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.860479:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.860480:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.860481:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.860482:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.860483:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.860484:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.860484:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860485:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.860487:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860488:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860489:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.860490:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.860491:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.860492:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.860495:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.860501:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.860502:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.860504:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.860505:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.860506:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.860507:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.860509:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.860509:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.860510:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.860511:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00002000:00000001:5.0:1496184949.860512:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.860513:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.860515:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.860520:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.860522:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.860523:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.860524:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.860525:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.860526:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.860527:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.860530:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57685 00000100:00000040:5.0:1496184949.860531:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.860533:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671704656 : -131872037846960 : ffff881026074050) 00000100:00000040:5.0:1496184949.860535:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.860540:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.860541:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.860543:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.860545:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.860546:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.860548:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.860549:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860550:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.860550:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.860552:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.860553:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.860554:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.860555:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.860555:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860557:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.860558:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.860559:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.860561:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e689000. 02000000:00000001:5.0:1496184949.860562:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.860563:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860565:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.860566:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.860567:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.860575:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.860578:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.860579:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.860581:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.860583:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.860584:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.860585:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.860587:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.860589:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.860589:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.860592:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.860593:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.860594:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.860594:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.860595:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.860597:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.860598:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.860599:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.860601:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.860623:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.860629:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860630:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.860633:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.860634:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.860635:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.860636:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.860637:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.860639:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.860640:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.860641:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.860646:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.860647:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.860674:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfed80-0x592def0cfed80 00000100:00000001:5.0:1496184949.860676:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.869880:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.869882:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.880868:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.880872:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.880873:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.891858:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.891920:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.891939:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.891942:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.891946:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.891947:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.891949:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.891950:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.891984:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.891986:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.891988:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.891989:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.891989:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.891991:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.891992:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.891994:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.891994:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.891995:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.891996:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.891998:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.892000:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184949.892001:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.892002:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.892003:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e689000. 02000000:00000001:5.0:1496184949.892004:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.892006:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.892008:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.892010:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.892016:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.892017:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.892018:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.892019:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.892021:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.892022:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.892028:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 31487us (31640us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.892033:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57685 00000100:00000040:5.0:1496184949.892035:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.892037:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.892037:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.892040:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.892045:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.892046:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.892048:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.892049:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.892050:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.892051:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.892053:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.892054:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.892054:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.892055:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074050. 00002000:00000001:5.0:1496184949.892056:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.892057:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.892059:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88103e674c80. 00000020:00000010:5.0:1496184949.892061:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.892062:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.892064:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.892065:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.892068:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.892073:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.892074:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.902491:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.902493:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.902495:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb850 x1568861104106896 msgsize 520 00000100:00100000:6.0:1496184949.902500:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.902501:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.902503:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.902534:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.902537:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.902539:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.902540:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.902541:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.902542:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.902545:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.902547:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106896 00000020:00000001:5.0:1496184949.902548:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.902549:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.902550:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.902551:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.902553:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.902555:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.902557:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.902557:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.902559:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.902561:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232e00. 00000020:00000010:5.0:1496184949.902563:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.902565:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.902565:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.902566:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.902569:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.902572:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdb850 x1568861104106896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.902577:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.902578:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902583:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.902585:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.902585:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.902586:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.902587:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.902588:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.902589:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.902591:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43416 00000100:00000040:5.0:1496184949.902593:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.902594:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683662928 : -131872025888688 : ffff881026bdb850) 00000100:00000040:5.0:1496184949.902597:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb850 x1568861104106896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.902602:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.902602:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.902604:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106896:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.902606:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106896 00000020:00000001:5.0:1496184949.902607:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.902609:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.902613:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.902615:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.902616:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.902617:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e689000. 02000000:00000001:5.0:1496184949.902619:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902620:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.902623:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.902624:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.902626:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.902628:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.902629:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.902631:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.902634:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.902636:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.902637:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.902638:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.902639:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.902640:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.902641:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.902643:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.902644:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.902645:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.902646:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.902648:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.902649:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.902652:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.902653:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.902655:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.902656:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.902657:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.902659:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.902661:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.902661:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.902662:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.902664:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.902666:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.902668:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.902669:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.902670:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.902672:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.902673:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.902674:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.902675:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.902677:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.902678:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.902679:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.902680:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.902681:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.902683:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.902684:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902686:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.902688:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.902692:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.902693:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.902695:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.902697:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.902698:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902699:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.902700:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.902701:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.902703:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.902704:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.902706:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902707:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.902708:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.902709:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.902710:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.902712:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.902712:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.902714:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.902716:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.902717:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.902718:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.902719:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.902721:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.902722:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.902723:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.902724:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.902725:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.902726:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.902727:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.902728:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.902729:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.902731:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106896 00010000:00000001:5.0:1496184949.902732:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.902734:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdb850 x1568861104106896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.902739:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.902740:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.902741:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.902744:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.902745:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.902747:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.902748:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.902749:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.902750:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902751:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.902753:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.902755:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106896, offset 0 00000100:00000001:5.0:1496184949.902770:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.902771:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.902773:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.902774:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.902775:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.902776:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.902777:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.902778:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb850 x1568861104106896/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.902784:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106896:12345-10.37.248.124@o2ib1:8 Request procesed in 181us (287us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.902788:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43416 00000100:00000040:5.0:1496184949.902790:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.902791:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.902792:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.902793:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232e00. 00000020:00000010:5.0:1496184949.902795:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.902797:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.902798:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.902799:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.902801:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.902805:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.902806:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.902861:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.902863:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.902864:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e689000. 02000000:00000001:7.0:1496184949.902867:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.902868:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.903328:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.903330:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.903332:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026074c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.903337:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.903338:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.903341:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.903372:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.903375:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.903376:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.903377:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.903378:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.903380:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.903382:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.903384:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.903385:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.903386:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.903387:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.903388:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.903390:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.903392:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.903394:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.903395:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.903397:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.903398:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232380. 00000020:00000010:5.0:1496184949.903400:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.903402:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.903404:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.903404:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.903405:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.903406:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.903408:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.903409:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.903410:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.903412:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.903412:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.903414:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.903414:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.903416:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.903417:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.903418:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.903418:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903419:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.903421:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903422:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903423:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.903424:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.903425:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.903426:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.903429:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.903435:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.903436:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.903438:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.903439:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.903440:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.903441:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.903443:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.903443:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.903444:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.903445:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00002000:00000001:5.0:1496184949.903446:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.903447:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.903449:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.903454:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.903456:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.903457:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.903458:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.903459:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.903460:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.903461:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.903463:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57686 00000100:00000040:5.0:1496184949.903465:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.903466:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671707728 : -131872037843888 : ffff881026074c50) 00000100:00000040:5.0:1496184949.903468:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.903474:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.903474:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.903476:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.903478:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.903479:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.903481:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.903482:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903483:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.903483:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.903485:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.903486:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.903487:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.903488:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.903488:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903489:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.903491:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.903492:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.903493:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184949.903495:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.903496:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903497:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.903498:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.903500:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.903504:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.903507:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.903507:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.903510:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.903511:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.903512:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.903513:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.903515:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.903517:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.903518:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.903520:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.903521:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.903522:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.903522:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.903523:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.903525:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.903526:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.903527:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.903529:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.903551:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.903557:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903558:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.903561:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.903562:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.903563:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.903564:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.903565:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.903566:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.903568:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.903569:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.903574:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.903575:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.903603:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfeda0-0x592def0cfeda0 00000100:00000001:5.0:1496184949.903605:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.905143:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.905146:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.905154:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.905170:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.905171:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.905176:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.905215:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.905221:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.905223:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.905227:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.905234:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.905236:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.905238:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.905271:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.905273:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.905275:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.905276:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.905277:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.905278:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.905280:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.905281:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.905282:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.905282:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.905283:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.905285:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.905287:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184949.905288:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.905289:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.905290:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184949.905291:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.905293:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.905295:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.905297:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.905302:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.905304:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.905304:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.905305:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.905306:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.905308:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.905314:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1839us (1980us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.905319:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57686 00000100:00000040:5.0:1496184949.905321:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.905322:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.905323:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.905326:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026074c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.905331:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.905332:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026074c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.905333:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.905334:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.905335:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.905337:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.905338:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.905339:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.905340:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.905340:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026074c50. 00002000:00000001:5.0:1496184949.905342:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.905343:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.905344:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232380. 00000020:00000010:5.0:1496184949.905346:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.905347:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.905349:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.905350:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.905352:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.905358:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.905359:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.934197:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.934199:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184949.934201:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdb450 x1568861104106928 msgsize 520 00000100:00100000:6.0:1496184949.934206:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.934207:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.934209:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.934241:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.934244:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.934245:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.934246:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.934247:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.934248:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.934251:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.934253:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106928 00000020:00000001:5.0:1496184949.934254:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.934256:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.934257:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.934258:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.934259:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.934261:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.934263:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.934264:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.934265:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.934267:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232300. 00000020:00000010:5.0:1496184949.934269:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.934270:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.934271:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.934272:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.934274:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.934277:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdb450 x1568861104106928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.934283:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.934283:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934288:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.934290:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.934291:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.934292:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.934292:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.934293:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.934294:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.934296:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43417 00000100:00000040:5.0:1496184949.934298:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.934299:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683661904 : -131872025889712 : ffff881026bdb450) 00000100:00000040:5.0:1496184949.934301:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdb450 x1568861104106928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.934307:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.934307:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.934309:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106928:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.934312:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106928 00000020:00000001:5.0:1496184949.934314:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.934316:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.934319:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.934321:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.934322:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.934324:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68ec00. 02000000:00000001:5.0:1496184949.934325:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934326:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.934329:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.934330:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.934332:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.934334:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.934335:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.934337:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.934340:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.934341:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.934342:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.934343:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.934344:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.934345:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.934346:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.934347:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.934349:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.934349:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.934350:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.934352:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.934353:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.934356:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.934357:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.934358:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.934361:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.934362:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.934364:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.934365:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.934366:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.934367:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.934369:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.934370:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.934372:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.934373:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.934374:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.934375:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.934377:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.934378:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.934379:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.934380:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.934381:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.934382:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.934383:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.934385:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.934386:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.934387:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934388:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.934391:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.934393:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.934395:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.934396:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.934398:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.934399:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934400:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.934401:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.934402:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.934404:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.934405:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.934407:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934408:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.934409:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.934410:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.934411:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.934412:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.934413:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.934414:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.934416:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.934417:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.934418:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.934419:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.934421:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.934422:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.934423:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.934424:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.934425:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.934426:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.934427:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.934428:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.934429:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.934430:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106928 00010000:00000001:5.0:1496184949.934432:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.934434:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdb450 x1568861104106928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.934439:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.934440:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.934441:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.934443:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.934445:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.934447:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.934448:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.934449:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.934450:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934451:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.934453:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.934455:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106928, offset 0 00000100:00000001:5.0:1496184949.934465:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.934466:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.934468:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.934469:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.934470:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.934471:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.934472:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.934473:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdb450 x1568861104106928/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.934479:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106928:12345-10.37.248.124@o2ib1:8 Request procesed in 171us (276us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.934483:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43417 00000100:00000040:5.0:1496184949.934485:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.934486:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.934487:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.934489:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232300. 00000020:00000010:5.0:1496184949.934490:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.934492:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.934493:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.934495:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.934496:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.934500:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.934500:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184949.934558:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184949.934560:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184949.934561:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68ec00. 02000000:00000001:6.0:1496184949.934564:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184949.934565:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184949.935050:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184949.935052:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184949.935055:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184949.935060:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184949.935061:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184949.935063:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.935093:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.935095:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.935097:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.935098:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.935099:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.935100:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.935103:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.935104:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.935106:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.935107:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.935108:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.935109:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.935111:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.935112:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.935114:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.935115:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.935117:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.935119:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232a80. 00000020:00000010:5.0:1496184949.935120:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.935123:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.935124:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.935125:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.935126:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.935127:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.935128:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.935129:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.935130:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.935132:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.935133:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.935134:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.935135:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.935136:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.935136:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.935137:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.935138:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935139:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.935140:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935141:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935142:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.935144:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.935145:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.935146:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.935149:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.935156:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.935156:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.935158:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.935159:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.935160:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.935161:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.935163:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.935163:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.935164:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.935165:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00002000:00000001:5.0:1496184949.935166:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.935167:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.935169:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.935174:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.935176:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.935177:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.935178:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.935178:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.935179:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.935180:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.935183:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57687 00000100:00000040:5.0:1496184949.935184:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.935186:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671700560 : -131872037851056 : ffff881026073050) 00000100:00000040:5.0:1496184949.935188:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.935193:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.935194:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.935195:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.935198:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.935199:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.935201:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.935201:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935203:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.935203:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.935205:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.935206:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.935207:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.935207:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.935208:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935209:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.935211:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.935212:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.935213:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b000. 02000000:00000001:5.0:1496184949.935214:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.935215:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935217:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.935218:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.935219:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.935224:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.935226:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.935227:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.935229:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.935231:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.935232:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.935233:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.935235:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.935237:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.935237:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.935239:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.935241:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.935241:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.935242:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.935243:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.935245:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.935246:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.935247:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.935249:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.935272:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.935278:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935279:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.935282:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.935283:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.935284:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.935285:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.935286:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.935288:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184949.935289:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.935290:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.935295:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.935296:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.935322:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfedc0-0x592def0cfedc0 00000100:00000001:5.0:1496184949.935325:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.937216:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.937219:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.937239:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.937246:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.937246:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184949.937251:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.937290:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.937304:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.937307:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.937311:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.937311:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.937314:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.937315:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.937348:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.937351:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.937353:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.937353:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.937354:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.937356:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.937357:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.937359:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.937359:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.937360:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.937361:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.937363:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.937365:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184949.937366:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.937367:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.937368:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b000. 02000000:00000001:5.0:1496184949.937369:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.937371:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.937373:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.937375:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.937380:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.937382:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.937383:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.937383:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.937385:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.937386:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.937392:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2198us (2336us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.937397:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57687 00000100:00000040:5.0:1496184949.937399:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.937400:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.937401:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.937404:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.937409:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.937410:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.937412:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.937413:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.937414:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.937415:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.937417:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.937418:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.937418:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.937419:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073050. 00002000:00000001:5.0:1496184949.937420:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.937421:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.937423:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232a80. 00000020:00000010:5.0:1496184949.937424:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.937426:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.937427:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.937429:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.937431:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.937436:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.937437:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.965628:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.965630:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.965632:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc050 x1568861104106960 msgsize 520 00000100:00100000:7.0:1496184949.965637:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.965638:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.965640:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.965671:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.965674:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.965675:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.965676:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.965678:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.965679:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.965681:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.965683:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106960 00000020:00000001:5.0:1496184949.965684:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.965685:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.965686:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.965688:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.965689:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.965691:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.965693:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.965693:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.965695:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.965697:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232580. 00000020:00000010:5.0:1496184949.965699:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.965701:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.965702:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.965702:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.965705:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.965708:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdc050 x1568861104106960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.965713:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.965714:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965718:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.965721:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.965721:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.965722:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.965723:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.965724:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.965725:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.965727:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43418 00000100:00000040:5.0:1496184949.965729:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.965730:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683664976 : -131872025886640 : ffff881026bdc050) 00000100:00000040:5.0:1496184949.965732:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc050 x1568861104106960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.965737:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.965738:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.965740:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106960:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.965742:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106960 00000020:00000001:5.0:1496184949.965743:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.965745:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.965748:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.965751:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.965752:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.965753:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68b000. 02000000:00000001:5.0:1496184949.965754:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965755:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.965758:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.965759:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.965761:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.965763:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.965764:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.965767:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.965769:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.965770:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.965771:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.965772:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.965773:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.965774:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.965775:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.965777:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.965778:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.965779:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.965780:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.965781:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.965782:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.965785:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.965786:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.965788:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.965789:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.965790:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.965792:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.965794:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.965794:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.965795:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.965797:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.965799:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.965800:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.965802:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.965802:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.965804:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.965806:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.965807:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.965808:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.965809:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.965810:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.965811:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.965812:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.965814:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.965815:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.965816:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965817:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.965820:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.965822:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.965823:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.965825:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.965826:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.965827:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965828:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.965830:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.965831:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.965833:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.965834:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.965835:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965836:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.965837:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.965838:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.965839:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.965840:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.965841:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.965843:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.965844:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.965846:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.965847:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.965848:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.965850:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.965851:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.965852:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.965853:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.965854:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.965855:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.965856:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.965857:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.965858:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.965859:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106960 00010000:00000001:5.0:1496184949.965861:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.965863:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdc050 x1568861104106960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.965868:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.965869:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.965870:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.965872:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.965874:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.965875:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.965876:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.965878:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.965878:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965879:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.965881:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.965883:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106960, offset 0 00000100:00000001:5.0:1496184949.965909:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.965911:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.965912:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.965914:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.965915:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.965916:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.965917:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.965919:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc050 x1568861104106960/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.965924:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106960:12345-10.37.248.124@o2ib1:8 Request procesed in 186us (291us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.965929:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43418 00000100:00000040:5.0:1496184949.965930:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.965931:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.965932:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.965934:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232580. 00000020:00000010:5.0:1496184949.965935:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.965937:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.965938:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.965940:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.965942:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.965946:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.965946:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.965998:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.966000:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.966002:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68b000. 02000000:00000001:7.0:1496184949.966005:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.966006:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.966434:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.966436:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.966439:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026073850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.966444:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.966445:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.966447:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.966478:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.966481:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.966482:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.966484:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.966485:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.966486:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.966488:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.966490:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.966491:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.966492:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.966493:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.966494:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.966496:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.966498:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.966500:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.966501:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.966502:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.966505:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff880850232480. 00000020:00000010:5.0:1496184949.966506:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.966508:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.966510:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.966511:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.966512:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.966513:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.966514:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.966515:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.966516:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.966518:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.966518:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.966520:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.966521:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.966522:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.966522:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.966523:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.966524:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966525:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.966526:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966527:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966528:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.966530:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.966530:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.966532:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.966535:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.966541:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.966541:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.966543:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.966544:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.966545:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.966547:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.966548:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.966549:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.966549:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.966550:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00002000:00000001:5.0:1496184949.966551:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.966552:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.966555:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.966559:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.966561:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.966562:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.966563:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.966564:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.966565:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.966566:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.966568:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57688 00000100:00000040:5.0:1496184949.966570:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.966571:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671702608 : -131872037849008 : ffff881026073850) 00000100:00000040:5.0:1496184949.966573:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.966578:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.966579:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.966581:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.966583:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.966584:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.966585:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.966586:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966587:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.966588:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.966589:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.966591:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.966592:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.966592:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.966593:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966594:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.966596:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.966597:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.966598:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68dc00. 02000000:00000001:5.0:1496184949.966600:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.966601:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966602:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.966603:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.966605:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.966609:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.966612:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.966612:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.966615:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.966616:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.966617:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.966618:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.966620:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.966622:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.966622:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.966624:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.966626:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.966626:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.966627:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.966628:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.966630:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.966630:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.966631:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.966634:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.966655:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.966661:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966662:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.966665:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.966666:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.966667:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.966668:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.966669:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.966670:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.966672:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.966673:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.966678:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.966679:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.966705:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfede0-0x592def0cfede0 00000100:00000001:5.0:1496184949.966707:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.968049:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.968052:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.968070:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184949.968074:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.968075:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184949.968080:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184949.968119:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184949.968126:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.968128:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184949.968140:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184949.968141:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.968143:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.968144:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184949.968178:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184949.968180:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184949.968182:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184949.968183:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184949.968183:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184949.968185:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.968186:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184949.968187:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184949.968188:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184949.968189:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184949.968190:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184949.968191:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184949.968193:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184949.968194:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184949.968195:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184949.968197:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68dc00. 02000000:00000001:5.0:1496184949.968198:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184949.968200:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184949.968201:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184949.968204:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184949.968209:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184949.968210:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184949.968211:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.968212:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.968213:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.968215:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184949.968221:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1641us (1780us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184949.968226:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57688 00000100:00000040:5.0:1496184949.968228:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.968229:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184949.968230:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.968233:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184949.968238:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.968239:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.968240:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.968241:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.968242:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.968244:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.968245:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.968246:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.968246:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.968247:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073850. 00002000:00000001:5.0:1496184949.968248:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.968250:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.968251:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff880850232480. 00000020:00000010:5.0:1496184949.968253:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.968254:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.968256:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.968257:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.968259:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184949.968265:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.968266:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.996454:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.996455:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184949.996458:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc450 x1568861104106992 msgsize 520 00000100:00100000:7.0:1496184949.996463:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.996464:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.996466:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.996500:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.996502:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.996504:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.996505:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.996506:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.996507:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.996509:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.996511:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104106992 00000020:00000001:5.0:1496184949.996513:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.996514:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.996515:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.996516:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.996518:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.996519:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.996521:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.996522:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.996524:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.996526:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88104b7cae00. 00000020:00000010:5.0:1496184949.996528:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184949.996529:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.996530:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184949.996531:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184949.996533:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184949.996536:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdc450 x1568861104106992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.996546:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184949.996546:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996551:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.996553:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.996554:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.996555:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.996555:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.996556:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.996557:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.996559:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43419 00000100:00000040:5.0:1496184949.996561:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.996562:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683666000 : -131872025885616 : ffff881026bdc450) 00000100:00000040:5.0:1496184949.996564:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc450 x1568861104106992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184969 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.996570:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.996570:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.996572:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106992:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184949.996574:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104106992 00000020:00000001:5.0:1496184949.996575:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184949.996577:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184949.996580:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184949.996583:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.996584:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.996585:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68dc00. 02000000:00000001:5.0:1496184949.996586:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996587:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.996590:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.996591:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184949.996593:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184949.996595:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184949.996596:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184949.996598:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184949 last 1496184949 00010000:00000001:5.0:1496184949.996601:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184949.996602:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184949.996603:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184949.996604:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184949.996605:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184949.996606:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184949.996607:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184949.996608:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184949.996610:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184949.996611:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184949.996612:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184949.996613:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184949.996614:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184949.996617:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.996618:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.996619:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.996621:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.996622:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.996624:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.996626:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184949.996626:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184949.996627:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184949.996629:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184949.996631:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184949.996632:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.996633:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184949.996634:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184949.996636:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184949.996637:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.996638:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.996639:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184949.996641:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184949.996642:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.996643:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184949.996644:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.996645:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.996647:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.996648:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996649:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184949.996652:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184949.996654:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184949.996655:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184949.996657:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.996658:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.996659:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996660:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.996662:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.996663:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.996664:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184949.996666:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184949.996667:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996668:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184949.996669:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184949.996670:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184949.996671:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184949.996672:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184949.996673:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184949.996674:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184949.996676:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184949.996677:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184949.996678:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184949.996679:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.996681:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.996682:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184949.996684:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.996684:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.996686:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.996686:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.996687:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.996688:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184949.996689:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184949.996691:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104106992 00010000:00000001:5.0:1496184949.996692:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184949.996694:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdc450 x1568861104106992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184949.996699:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184949.996700:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184949.996701:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=94 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184949.996703:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184949.996705:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184949.996706:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.996708:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184949.996709:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184949.996710:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996711:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184949.996712:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184949.996714:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104106992, offset 0 00000100:00000001:5.0:1496184949.996725:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.996726:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184949.996727:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184949.996728:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184949.996729:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184949.996730:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.996731:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184949.996733:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc450 x1568861104106992/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184969 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184949.996739:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104106992:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (279us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184949.996743:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43419 00000100:00000040:5.0:1496184949.996745:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184949.996746:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184949.996747:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184949.996748:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88104b7cae00. 00000020:00000010:5.0:1496184949.996750:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184949.996751:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184949.996753:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184949.996754:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184949.996756:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184949.996759:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.996760:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184949.996828:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184949.996831:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184949.996833:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68dc00. 02000000:00000001:7.0:1496184949.996838:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184949.996839:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184949.997330:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184949.997332:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184949.997342:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184949.997347:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184949.997348:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184949.997350:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184949.997380:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.997382:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.997384:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184949.997385:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184949.997386:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184949.997387:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184949.997389:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184949.997391:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184949.997393:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184949.997394:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184949.997395:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184949.997396:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184949.997398:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184949.997399:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184949.997401:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.997402:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184949.997404:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184949.997406:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88104b7caa80. 00000020:00000010:5.0:1496184949.997408:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184949.997410:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184949.997411:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184949.997412:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184949.997413:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184949.997414:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.997415:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.997417:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.997418:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184949.997419:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184949.997420:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.997421:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184949.997422:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.997423:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.997424:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.997425:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184949.997426:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997427:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184949.997428:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997429:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997430:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.997432:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184949.997432:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.997434:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184949.997437:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184949.997443:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184949.997444:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184949.997446:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184949.997446:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184949.997447:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.997449:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184949.997450:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184949.997451:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184949.997451:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184949.997452:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00002000:00000001:5.0:1496184949.997453:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.997455:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.997457:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.997462:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184949.997464:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184949.997465:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184949.997465:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184949.997466:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184949.997467:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184949.997468:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184949.997470:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57689 00000100:00000040:5.0:1496184949.997472:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184949.997473:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671695440 : -131872037856176 : ffff881026071c50) 00000100:00000040:5.0:1496184949.997476:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/0 e 0 to 0 dl 1496184992 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184949.997481:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184949.997482:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184949.997483:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184949.997485:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184949.997486:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184949.997488:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184949.997489:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997490:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184949.997491:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184949.997492:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184949.997493:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184949.997494:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184949.997495:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184949.997496:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997497:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184949.997498:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184949.997499:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184949.997501:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68d800. 02000000:00000001:5.0:1496184949.997502:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.997503:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997504:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184949.997505:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184949.997507:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184949.997512:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184949.997514:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184949.997515:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184949.997517:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184949.997519:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184949.997520:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184949.997521:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184949.997522:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184949.997524:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184949.997525:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184949.997527:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184949.997528:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184949.997529:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184949.997529:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184949.997530:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184949.997532:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184949.997533:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184949.997534:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184949.997536:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184949.997560:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184949.997565:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997567:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184949.997569:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.997571:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184949.997572:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184949.997573:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184949.997574:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184949.997575:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184949.997577:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184949.997578:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184949.997582:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184949.997583:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184949.997610:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfee00-0x592def0cfee00 00000100:00000001:5.0:1496184949.997612:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184949.999226:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184949.999229:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.010357:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.010362:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.010363:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.021354:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.021422:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.021441:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.021444:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.021447:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.021448:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.021451:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.021452:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.021485:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.021488:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.021490:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.021491:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.021491:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.021493:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.021494:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.021495:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.021496:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.021497:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.021498:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.021499:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.021501:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.021502:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.021503:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.021504:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68d800. 02000000:00000001:5.0:1496184950.021506:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.021507:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.021509:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.021512:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.021517:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.021518:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.021519:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.021520:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.021521:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.021523:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.021529:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24047us (24185us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.021534:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57689 00000100:00000040:5.0:1496184950.021536:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.021537:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.021538:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.021541:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:737/0 lens 608/448 e 0 to 0 dl 1496184992 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.021546:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.021547:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.021549:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.021550:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.021551:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.021552:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.021554:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.021554:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.021555:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.021556:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071c50. 00002000:00000001:5.0:1496184950.021557:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.021558:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.021560:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88104b7caa80. 00000020:00000010:5.0:1496184950.021561:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.021563:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.021564:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.021566:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.021568:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.021573:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.021574:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.032194:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.032196:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.032198:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdc850 x1568861104107024 msgsize 520 00000100:00100000:6.0:1496184950.032203:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.032204:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.032206:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.032239:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.032241:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.032243:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.032244:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.032245:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.032247:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.032249:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.032251:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107024 00000020:00000001:5.0:1496184950.032253:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.032253:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.032255:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.032256:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.032257:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.032259:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.032261:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.032262:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.032263:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.032266:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7480. 00000020:00000010:5.0:1496184950.032267:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.032269:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.032270:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.032271:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.032273:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.032276:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdc850 x1568861104107024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.032281:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.032282:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032287:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.032289:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.032290:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.032291:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.032291:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.032292:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.032293:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.032296:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43420 00000100:00000040:5.0:1496184950.032297:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.032298:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683667024 : -131872025884592 : ffff881026bdc850) 00000100:00000040:5.0:1496184950.032301:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdc850 x1568861104107024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.032306:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.032307:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.032308:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107024:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.032311:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107024 00000020:00000001:5.0:1496184950.032312:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.032314:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.032317:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.032319:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.032320:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.032322:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68d800. 02000000:00000001:5.0:1496184950.032323:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032324:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.032326:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.032328:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.032330:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.032331:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.032333:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.032335:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.032337:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.032339:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.032339:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.032340:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.032341:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.032342:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.032343:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.032345:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.032346:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.032347:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.032348:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.032349:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.032351:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.032354:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.032355:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.032356:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.032358:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.032359:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.032361:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.032362:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.032363:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.032364:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.032366:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.032367:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.032368:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.032369:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.032370:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.032372:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.032373:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.032374:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.032375:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.032377:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.032378:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.032379:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.032380:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.032381:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.032383:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.032384:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032385:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.032388:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.032391:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.032392:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.032394:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.032395:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.032396:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032397:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.032399:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.032400:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.032401:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.032402:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.032404:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032405:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.032406:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.032407:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.032408:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.032409:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.032410:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.032411:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.032413:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.032414:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.032415:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.032416:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.032418:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.032419:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.032420:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.032421:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.032422:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.032423:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.032424:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.032425:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.032426:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.032427:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107024 00010000:00000001:5.0:1496184950.032429:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.032431:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdc850 x1568861104107024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.032436:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.032437:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.032438:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.032441:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.032442:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.032444:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.032445:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.032446:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.032447:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032448:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.032450:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.032452:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107024, offset 0 00000100:00000001:5.0:1496184950.032462:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.032463:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.032465:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.032466:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.032467:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.032468:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.032469:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.032470:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdc850 x1568861104107024/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.032476:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107024:12345-10.37.248.124@o2ib1:8 Request procesed in 169us (276us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.032481:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43420 00000100:00000040:5.0:1496184950.032482:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.032483:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.032484:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.032486:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7480. 00000020:00000010:5.0:1496184950.032487:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.032489:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.032490:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.032492:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.032494:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.032497:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.032497:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.032566:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.032569:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.032571:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68d800. 02000000:00000001:6.0:1496184950.032576:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.032577:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.033006:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.033008:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.033017:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026073450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.033022:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.033023:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.033025:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.033057:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.033060:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.033061:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.033062:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.033064:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.033065:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.033067:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.033069:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.033070:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.033071:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.033072:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.033074:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.033075:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.033077:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.033079:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.033080:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.033082:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.033083:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7180. 00000020:00000010:5.0:1496184950.033085:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.033087:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.033088:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.033089:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.033090:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.033091:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.033092:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.033094:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.033095:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.033096:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.033097:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.033098:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.033099:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.033100:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.033101:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.033102:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.033103:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033104:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.033105:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033106:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033107:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.033109:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.033109:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.033111:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.033114:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.033120:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.033121:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.033123:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.033123:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.033124:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.033126:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.033127:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.033128:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.033129:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.033129:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00002000:00000001:5.0:1496184950.033131:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.033132:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.033134:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.033138:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.033141:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.033141:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.033142:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.033143:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.033144:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.033145:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.033147:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57690 00000100:00000040:5.0:1496184950.033149:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.033150:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671701584 : -131872037850032 : ffff881026073450) 00000100:00000040:5.0:1496184950.033153:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.033158:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.033159:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.033161:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.033163:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.033164:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.033165:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.033166:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033167:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.033168:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.033169:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.033170:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.033171:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.033172:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.033173:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033174:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.033176:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.033176:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.033178:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.033179:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.033180:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033182:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.033182:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.033184:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.033189:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.033191:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.033192:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.033194:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.033196:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.033197:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.033198:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.033199:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.033201:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.033202:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.033204:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.033205:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.033206:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.033207:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.033208:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.033210:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.033210:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.033211:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.033214:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.033235:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.033241:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033243:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.033245:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.033246:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.033248:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.033248:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.033250:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.033251:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.033252:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.033254:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.033258:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.033259:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.033285:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfee20-0x592def0cfee20 00000100:00000001:5.0:1496184950.033287:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.034948:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.034951:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.046004:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.046009:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.046010:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.057039:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.057099:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.057118:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.057120:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.057124:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.057125:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.057127:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00000100:00000001:1.0:1496184950.057127:0:21159:0:(events.c:294:request_in_callback()) Process entered 00002000:00000001:5.0:1496184950.057129:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00000200:1.0:1496184950.057129:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184950.057132:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89450 x1568861104107056 msgsize 520 00000100:00100000:1.0:1496184950.057139:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184950.057140:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184950.057143:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.057149:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.057152:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.057156:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184950.057158:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.057160:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000001:00000001:5.0:1496184950.057162:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057162:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.057165:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 02000000:00000001:3.0:1496184950.057166:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.057167:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.057168:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.057168:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.057170:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1496184950.057170:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107056 00000020:00000001:5.0:1496184950.057171:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.057172:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.057173:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000020:00000001:3.0:1496184950.057173:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000100:00000001:5.0:1496184950.057174:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:3.0:1496184950.057174:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000040:5.0:1496184950.057175:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.057176:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000001:3.0:1496184950.057177:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000010:5.0:1496184950.057178:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.057179:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 00000020:00000040:3.0:1496184950.057179:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 02000000:00000001:5.0:1496184950.057180:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 00000020:00000001:3.0:1496184950.057181:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 02000000:00000010:5.0:1496184950.057182:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.057183:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:00000001:3.0:1496184950.057183:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:02000400:5.0:1496184950.057185:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.057186:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184950.057187:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.057188:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00010000:00000080:5.0:1496184950.057189:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.057194:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000020:00000010:3.0:1496184950.057194:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88085de9a000. 00010000:00000001:5.0:1496184950.057196:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.057196:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.057197:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:3.0:1496184950.057197:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8600. 00000020:00000001:5.0:1496184950.057198:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.057200:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184950.057200:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184950.057203:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184950.057204:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184950.057205:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00100000:5.0:1496184950.057206:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24047us (24187us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.057211:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57690 00000100:00000001:3.0:1496184950.057211:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000040:5.0:1496184950.057213:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.057214:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.057215:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000200:3.0:1496184950.057216:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b89450 x1568861104107056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00002000:00100000:5.0:1496184950.057218:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026073450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.057222:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184950.057223:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.057224:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.057224:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026073450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.057226:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000100:00000001:3.0:1496184950.057227:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.057228:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.057229:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.057230:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00004000:00000001:3.0:1496184950.057231:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00010000:00000001:5.0:1496184950.057232:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.057232:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00004000:00000001:3.0:1496184950.057232:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00002000:00000001:5.0:1496184950.057233:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00004000:00000001:3.0:1496184950.057233:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00002000:00010000:5.0:1496184950.057234:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026073450. 00004000:00000001:3.0:1496184950.057234:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00002000:00000001:5.0:1496184950.057235:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057235:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.057236:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:3.0:1496184950.057237:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000020:00000010:5.0:1496184950.057238:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7180. 00000020:00000010:5.0:1496184950.057240:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000100:00100000:3.0:1496184950.057240:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6109 00000020:00000010:5.0:1496184950.057241:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.057243:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000040:3.0:1496184950.057243:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.057244:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184950.057244:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736385104 : -131906973166512 : ffff880803b89450) 00000100:00000200:5.0:1496184950.057247:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00000040:3.0:1496184950.057247:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89450 x1568861104107056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00004000:00000001:5.0:1496184950.057252:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.057253:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184950.057253:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.057253:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184950.057256:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107056:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184950.057258:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107056 00000020:00000001:3.0:1496184950.057260:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184950.057265:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184950.057273:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184950.057277:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184950.057279:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184950.057282:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fd800. 02000000:00000001:3.0:1496184950.057283:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057286:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.057292:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184950.057294:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184950.057304:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184950.057307:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184950.057309:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184950.057312:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:3.0:1496184950.057315:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.057318:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.057321:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.057323:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.057325:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.057326:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.057328:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.057330:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.057332:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.057333:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.057335:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.057336:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.057338:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.057343:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.057345:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.057346:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.057349:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.057350:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.057353:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.057355:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.057356:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.057357:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.057359:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.057362:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.057363:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.057365:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.057366:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.057368:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.057370:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.057371:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.057373:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.057374:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.057377:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.057378:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.057379:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.057381:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.057383:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.057385:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057386:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.057390:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.057392:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.057395:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.057397:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.057398:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.057400:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057401:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.057402:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.057404:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.057406:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.057408:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.057410:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057411:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.057414:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.057415:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.057416:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.057418:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.057420:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.057422:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.057424:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.057425:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.057427:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.057429:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.057432:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.057434:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.057435:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.057436:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.057438:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.057439:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.057441:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.057442:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.057445:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.057447:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107056 00010000:00000001:3.0:1496184950.057449:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.057452:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b89450 x1568861104107056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.057459:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.057460:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.057461:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.057465:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.057466:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.057468:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.057469:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.057478:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.057480:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057481:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.057483:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.057487:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107056, offset 0 00000100:00000001:3.0:1496184950.057503:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.057504:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.057506:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.057507:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.057509:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.057510:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.057511:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.057513:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89450 x1568861104107056/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.057519:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107056:12345-10.37.248.124@o2ib1:8 Request procesed in 265us (385us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.057524:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6109 00000100:00000040:3.0:1496184950.057525:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.057527:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.057528:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.057531:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8600. 00000020:00000010:3.0:1496184950.057533:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184950.057535:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88085de9a000. 00000020:00000040:3.0:1496184950.057537:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.057539:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.057541:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.057547:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.057548:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.057866:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.057881:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.057883:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fd800. 02000000:00000001:7.0:1496184950.057886:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.057887:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.058308:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.058310:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.058312:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.058317:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.058318:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.058321:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.058354:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.058357:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.058359:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.058360:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.058361:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.058362:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.058364:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.058366:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.058368:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.058369:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.058370:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.058372:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.058373:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.058375:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.058378:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.058379:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.058380:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.058383:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7780. 00000020:00000010:5.0:1496184950.058384:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.058387:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.058388:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.058389:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.058390:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.058391:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.058393:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.058394:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.058395:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.058397:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.058397:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.058399:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.058400:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.058401:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.058401:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.058402:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.058403:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058404:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.058405:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058407:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058408:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.058409:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.058410:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.058411:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.058414:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.058425:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.058426:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.058428:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.058428:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.058429:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.058431:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.058432:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.058433:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.058433:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.058434:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00002000:00000001:5.0:1496184950.058435:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.058437:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.058439:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.058443:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.058445:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.058446:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.058447:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.058448:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.058449:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.058450:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.058453:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57691 00000100:00000040:5.0:1496184950.058454:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.058455:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671697488 : -131872037854128 : ffff881026072450) 00000100:00000040:5.0:1496184950.058458:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.058463:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.058464:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.058465:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.058468:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.058469:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.058471:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.058472:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058473:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.058473:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.058475:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.058476:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.058477:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.058478:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.058479:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058480:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.058482:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.058482:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.058484:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.058485:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.058487:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058488:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.058489:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.058491:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.058495:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.058498:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.058498:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.058501:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.058502:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.058504:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.058504:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.058506:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.058508:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.058509:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.058511:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.058512:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.058513:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.058514:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.058515:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.058517:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.058517:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.058518:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.058521:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.058543:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.058549:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058550:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.058553:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.058554:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.058555:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.058556:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.058557:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.058559:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.058560:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.058562:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.058566:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.058567:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.058594:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfee40-0x592def0cfee40 00000100:00000001:5.0:1496184950.058596:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.069927:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.069930:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.080927:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.080932:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.080932:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.091904:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.091975:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.091982:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.091984:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.091988:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.091989:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.091991:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.091993:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.092026:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.092028:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.092030:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.092031:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.092032:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.092033:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.092035:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.092036:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.092037:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.092038:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.092039:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.092040:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.092042:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.092044:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.092045:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.092046:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.092047:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.092049:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.092050:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.092053:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.092058:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.092060:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.092061:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.092061:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.092063:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.092064:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.092070:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33606us (33756us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.092075:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57691 00000100:00000040:5.0:1496184950.092077:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.092079:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.092080:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.092082:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.092088:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.092089:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.092090:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.092091:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.092092:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.092094:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.092095:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.092096:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.092097:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.092097:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072450. 00002000:00000001:5.0:1496184950.092099:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.092100:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.092101:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7780. 00000020:00000010:5.0:1496184950.092103:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.092105:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.092106:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.092108:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.092110:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.092115:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.092116:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.102367:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.102369:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.102371:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdcc50 x1568861104107088 msgsize 520 00000100:00100000:7.0:1496184950.102376:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.102377:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.102380:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.102413:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.102416:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.102417:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.102418:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.102419:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.102420:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.102423:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.102425:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107088 00000020:00000001:5.0:1496184950.102426:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.102427:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.102428:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.102429:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.102431:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.102432:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.102434:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.102435:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.102437:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.102439:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7380. 00000020:00000010:5.0:1496184950.102440:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.102442:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.102443:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.102444:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.102446:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.102449:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdcc50 x1568861104107088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.102454:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.102455:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102460:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.102462:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.102463:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.102464:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.102464:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.102465:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.102466:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.102469:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43421 00000100:00000040:5.0:1496184950.102470:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.102471:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683668048 : -131872025883568 : ffff881026bdcc50) 00000100:00000040:5.0:1496184950.102474:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdcc50 x1568861104107088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.102479:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.102480:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.102482:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107088:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.102484:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107088 00000020:00000001:5.0:1496184950.102485:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.102487:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.102490:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.102493:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.102494:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.102495:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.102496:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102497:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.102500:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.102502:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.102504:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.102505:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.102507:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.102509:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.102512:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.102513:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.102514:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.102515:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.102516:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.102517:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.102518:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.102520:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.102521:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.102522:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.102524:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.102525:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.102526:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.102529:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.102530:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.102531:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.102533:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.102534:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.102536:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.102537:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.102538:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.102539:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.102541:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.102543:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.102544:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.102545:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.102546:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.102548:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.102550:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.102551:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.102552:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.102553:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.102555:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.102556:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.102557:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.102558:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.102560:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.102561:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102562:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.102565:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.102567:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.102568:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.102570:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.102571:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.102573:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102574:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.102575:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.102577:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.102578:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.102579:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.102581:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102582:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.102583:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.102584:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.102585:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.102587:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.102587:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.102589:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.102591:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.102592:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.102593:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.102594:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.102596:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.102597:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.102599:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.102599:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.102600:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.102601:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.102602:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.102603:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.102604:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.102606:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107088 00010000:00000001:5.0:1496184950.102607:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.102609:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdcc50 x1568861104107088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.102614:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.102615:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.102616:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.102619:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.102620:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.102622:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.102623:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.102624:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.102625:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102626:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.102628:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.102630:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107088, offset 0 00000100:00000001:5.0:1496184950.102645:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.102646:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.102648:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.102649:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.102650:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.102651:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.102652:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.102654:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdcc50 x1568861104107088/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.102660:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107088:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (287us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.102664:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43421 00000100:00000040:5.0:1496184950.102665:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.102667:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.102667:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.102669:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7380. 00000020:00000010:5.0:1496184950.102670:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.102672:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.102673:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.102675:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.102677:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.102680:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.102681:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.102714:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.102716:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.102718:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68f800. 02000000:00000001:7.0:1496184950.102720:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.102721:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.103168:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.103170:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.103172:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072c50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.103177:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.103179:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.103181:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.103211:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.103214:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.103216:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.103217:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.103218:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.103219:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.103221:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.103223:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.103225:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.103225:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.103227:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.103228:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.103229:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.103231:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.103233:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.103234:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.103236:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.103238:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7a00. 00000020:00000010:5.0:1496184950.103239:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.103241:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.103243:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.103244:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.103245:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.103246:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.103247:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.103248:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.103249:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.103251:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.103252:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.103253:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.103254:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.103255:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.103255:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.103256:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.103257:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103258:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.103259:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103260:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103262:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.103263:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.103264:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.103265:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.103268:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.103274:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.103275:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.103277:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.103277:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.103278:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.103280:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.103281:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.103282:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.103283:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.103283:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00002000:00000001:5.0:1496184950.103285:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.103286:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.103288:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.103293:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.103295:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.103295:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.103297:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.103297:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.103298:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.103299:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.103302:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57692 00000100:00000040:5.0:1496184950.103303:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.103304:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671699536 : -131872037852080 : ffff881026072c50) 00000100:00000040:5.0:1496184950.103307:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.103312:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.103313:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.103315:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.103317:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.103318:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.103320:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.103321:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103322:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.103322:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.103324:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.103325:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.103326:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.103327:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.103327:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103328:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.103330:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.103331:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.103333:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.103334:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.103335:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103336:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.103337:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.103339:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.103343:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.103346:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.103347:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.103349:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.103351:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.103352:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.103353:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.103354:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.103356:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.103357:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.103359:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.103360:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.103361:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.103361:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.103362:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.103364:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.103365:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.103366:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.103368:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.103390:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.103395:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103397:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.103399:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.103401:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.103402:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.103403:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.103404:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.103405:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184950.103407:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.103408:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.103413:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.103414:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.103441:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfee60-0x592def0cfee60 00000100:00000001:5.0:1496184950.103443:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.105146:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.105149:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.105170:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.105175:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.105175:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.105181:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.105219:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.105226:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.105228:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.105232:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.105233:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.105235:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.105236:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.105269:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.105272:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.105274:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.105275:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.105275:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.105277:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.105278:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.105279:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.105280:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.105280:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.105282:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.105283:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.105285:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.105286:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.105287:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.105288:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.105289:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.105291:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.105293:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.105296:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.105301:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.105302:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.105303:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.105303:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.105305:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.105307:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.105312:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1999us (2138us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.105317:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57692 00000100:00000040:5.0:1496184950.105319:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.105320:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.105321:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.105324:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.105330:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.105330:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.105332:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.105333:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.105334:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.105335:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.105337:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.105338:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.105338:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.105339:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072c50. 00002000:00000001:5.0:1496184950.105340:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.105341:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.105343:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7a00. 00000020:00000010:5.0:1496184950.105345:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.105346:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.105348:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.105349:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.105351:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.105357:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.105358:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.133963:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.133965:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.133967:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd050 x1568861104107120 msgsize 520 00000100:00100000:6.0:1496184950.133972:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.133973:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.133976:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.134008:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.134010:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.134012:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.134013:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.134014:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.134015:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.134017:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.134019:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107120 00000020:00000001:5.0:1496184950.134021:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.134022:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.134023:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.134024:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.134026:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.134027:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.134029:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.134030:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.134032:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.134034:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7980. 00000020:00000010:5.0:1496184950.134035:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.134037:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.134038:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.134039:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.134041:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.134044:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdd050 x1568861104107120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.134049:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.134050:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134055:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.134057:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.134058:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.134059:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.134060:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.134061:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.134062:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.134064:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43422 00000100:00000040:5.0:1496184950.134065:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.134067:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683669072 : -131872025882544 : ffff881026bdd050) 00000100:00000040:5.0:1496184950.134069:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd050 x1568861104107120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.134074:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.134075:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.134077:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107120:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.134079:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107120 00000020:00000001:5.0:1496184950.134080:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.134082:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.134085:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.134087:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.134088:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.134090:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.134091:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134092:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.134095:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.134096:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.134098:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.134099:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.134101:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.134103:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.134105:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.134107:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.134108:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.134109:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.134110:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.134111:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.134112:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.134113:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.134114:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.134115:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.134116:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.134118:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.134119:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.134122:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.134123:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.134124:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.134126:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.134126:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.134129:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.134130:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.134131:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.134132:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.134134:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.134135:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.134137:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134138:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.134139:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.134140:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.134142:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134143:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.134144:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.134145:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.134146:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.134147:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.134148:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.134150:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.134151:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.134153:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134154:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.134156:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.134158:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.134160:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.134162:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.134163:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.134164:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134165:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.134166:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.134168:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.134169:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.134170:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.134171:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134173:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.134174:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.134174:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.134175:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.134177:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.134178:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.134179:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.134180:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.134182:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.134183:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.134184:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134186:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.134187:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.134188:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.134189:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.134190:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.134191:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.134192:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134193:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.134194:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.134195:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107120 00010000:00000001:5.0:1496184950.134197:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.134199:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdd050 x1568861104107120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.134204:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.134205:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.134206:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.134208:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.134210:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.134211:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.134213:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.134214:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.134215:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134216:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.134217:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.134219:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107120, offset 0 00000100:00000001:5.0:1496184950.134230:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134231:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.134232:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.134233:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.134235:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.134235:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134236:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.134238:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd050 x1568861104107120/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.134244:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107120:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (275us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.134248:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43422 00000100:00000040:5.0:1496184950.134250:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.134251:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.134252:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.134254:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7980. 00000020:00000010:5.0:1496184950.134255:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.134257:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.134258:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.134260:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.134261:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.134265:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.134265:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.134321:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.134323:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.134324:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68e400. 02000000:00000001:7.0:1496184950.134327:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.134328:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.134695:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.134696:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.134699:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.134704:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.134705:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.134707:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.134738:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.134740:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.134742:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.134743:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.134744:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.134745:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.134747:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.134749:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.134751:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.134752:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.134753:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.134754:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.134756:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.134757:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.134759:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.134760:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.134762:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.134764:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7b00. 00000020:00000010:5.0:1496184950.134765:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.134767:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.134769:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.134769:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.134771:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.134772:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.134773:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.134774:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.134775:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.134777:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.134778:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.134779:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.134780:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.134781:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.134782:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.134783:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.134783:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134784:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.134785:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134786:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134787:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134789:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.134790:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134791:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.134794:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.134800:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.134801:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.134803:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.134803:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.134804:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.134806:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.134807:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.134808:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.134809:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.134809:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00002000:00000001:5.0:1496184950.134811:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134812:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134814:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134818:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.134820:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.134821:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.134822:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.134823:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.134824:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.134825:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.134827:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57693 00000100:00000040:5.0:1496184950.134829:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.134830:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671698512 : -131872037853104 : ffff881026072850) 00000100:00000040:5.0:1496184950.134833:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.134838:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.134838:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.134840:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.134842:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.134843:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.134845:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.134846:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134847:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.134848:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.134849:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.134850:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.134851:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.134852:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.134853:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134854:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.134855:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.134856:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.134858:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.134859:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134860:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134861:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.134862:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.134864:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.134868:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.134871:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.134872:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.134874:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.134876:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.134877:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.134878:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.134879:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.134881:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.134882:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.134884:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.134885:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.134886:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.134887:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.134888:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.134903:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.134903:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.134904:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.134907:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.134930:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.134936:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134937:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.134940:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134942:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.134943:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.134943:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.134945:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.134946:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184950.134948:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.134949:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.134954:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.134955:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.134981:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfee80-0x592def0cfee80 00000100:00000001:5.0:1496184950.134983:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.136403:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.136406:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.136425:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.136432:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.136432:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.136437:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.136475:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.136493:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.136495:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.136499:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.136500:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.136502:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.136503:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.136537:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.136539:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.136541:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.136542:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.136542:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.136544:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.136545:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.136547:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.136547:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.136548:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.136549:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.136551:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.136552:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.136554:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.136555:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.136556:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.136557:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.136559:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.136560:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.136563:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.136568:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.136570:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.136570:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.136571:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.136572:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.136574:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.136580:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1741us (1880us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.136585:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57693 00000100:00000040:5.0:1496184950.136587:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.136588:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.136589:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.136592:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.136597:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.136598:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.136600:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.136601:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.136602:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.136603:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.136605:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.136605:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.136606:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.136607:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072850. 00002000:00000001:5.0:1496184950.136608:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.136609:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.136611:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7b00. 00000020:00000010:5.0:1496184950.136612:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.136614:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.136615:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.136617:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.136619:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.136624:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.136625:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.165667:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.165669:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.165671:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd450 x1568861104107152 msgsize 520 00000100:00100000:7.0:1496184950.165676:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.165677:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.165679:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.165710:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.165712:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.165714:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.165715:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.165717:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.165718:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.165720:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.165722:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107152 00000020:00000001:5.0:1496184950.165723:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.165724:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.165725:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.165727:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.165728:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.165730:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.165732:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.165733:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.165734:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.165736:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7400. 00000020:00000010:5.0:1496184950.165738:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.165739:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.165740:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.165741:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.165743:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.165746:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdd450 x1568861104107152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.165751:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.165752:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165757:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.165759:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.165760:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.165761:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.165762:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.165762:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.165763:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.165766:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43423 00000100:00000040:5.0:1496184950.165767:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.165769:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683670096 : -131872025881520 : ffff881026bdd450) 00000100:00000040:5.0:1496184950.165771:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd450 x1568861104107152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.165776:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.165777:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.165779:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107152:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.165781:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107152 00000020:00000001:5.0:1496184950.165782:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.165784:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.165787:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.165790:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.165791:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.165792:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.165793:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165794:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.165797:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.165798:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.165800:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.165802:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.165803:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.165805:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.165808:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.165809:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.165810:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.165811:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.165812:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.165813:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.165814:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.165815:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.165816:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.165817:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.165819:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.165820:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.165821:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.165824:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.165825:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.165826:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.165828:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.165829:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.165831:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.165832:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.165833:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.165834:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.165836:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.165837:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.165839:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.165840:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.165841:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.165842:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.165844:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.165845:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.165846:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.165847:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.165848:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.165849:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.165850:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.165852:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.165853:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.165854:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165855:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.165858:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.165860:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.165862:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.165863:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.165865:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.165866:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165867:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.165868:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.165869:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.165871:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.165872:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.165873:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165874:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.165875:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.165876:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.165877:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.165879:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.165879:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.165880:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.165882:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.165883:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.165884:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.165886:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.165887:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.165888:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.165901:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.165902:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.165903:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.165904:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.165905:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.165906:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.165907:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.165908:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107152 00010000:00000001:5.0:1496184950.165910:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.165912:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdd450 x1568861104107152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.165917:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.165926:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.165927:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.165929:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.165931:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.165932:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.165933:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.165935:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.165936:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165937:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.165938:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.165940:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107152, offset 0 00000100:00000001:5.0:1496184950.165951:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.165952:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.165953:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.165954:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.165955:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.165956:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.165957:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.165959:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd450 x1568861104107152/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.165964:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107152:12345-10.37.248.124@o2ib1:8 Request procesed in 187us (291us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.165969:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43423 00000100:00000040:5.0:1496184950.165970:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.165971:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.165972:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.165974:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7400. 00000020:00000010:5.0:1496184950.165976:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.165977:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.165987:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.165988:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.165990:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.165994:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.165994:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.166063:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.166065:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.166067:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68cc00. 02000000:00000001:7.0:1496184950.166070:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.166071:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.166443:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.166445:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.166447:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026072050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.166453:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.166454:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.166456:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.166488:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.166491:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.166492:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.166493:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.166494:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.166495:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.166498:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.166500:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.166501:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.166502:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.166503:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.166504:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.166506:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.166508:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.166510:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.166511:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.166512:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.166514:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7e00. 00000020:00000010:5.0:1496184950.166516:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.166518:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.166520:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.166521:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.166522:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.166523:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.166524:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.166525:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.166526:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.166528:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.166528:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.166530:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.166531:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.166532:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.166532:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.166533:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.166534:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166535:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.166537:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166538:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166539:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.166540:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.166541:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.166542:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.166545:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.166551:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.166552:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.166554:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.166555:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.166556:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.166557:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.166559:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.166560:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.166560:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.166561:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00002000:00000001:5.0:1496184950.166562:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.166563:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.166565:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.166570:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.166572:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.166573:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.166574:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.166574:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.166575:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.166576:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.166578:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57694 00000100:00000040:5.0:1496184950.166580:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.166581:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671696464 : -131872037855152 : ffff881026072050) 00000100:00000040:5.0:1496184950.166584:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.166589:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.166590:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.166592:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.166594:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.166595:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.166597:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.166597:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166598:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.166599:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.166601:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.166602:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.166603:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.166603:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.166604:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166605:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.166607:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.166608:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.166609:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.166610:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.166612:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166613:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.166614:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.166616:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.166620:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.166622:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.166623:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.166625:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.166627:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.166628:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.166629:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.166630:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.166632:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.166633:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.166635:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.166637:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.166637:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.166638:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.166639:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.166641:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.166641:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.166642:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.166645:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.166666:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.166672:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166674:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.166676:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.166678:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.166679:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.166680:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.166681:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.166682:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.166683:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.166685:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.166689:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.166690:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.166717:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfeea0-0x592def0cfeea0 00000100:00000001:5.0:1496184950.166719:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.168189:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.168192:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.168212:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.168216:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.168217:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.168222:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.168267:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.168280:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.168282:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.168285:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.168286:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.168288:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.168290:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.168323:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.168325:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.168327:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.168329:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.168329:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.168331:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.168332:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.168333:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.168334:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.168335:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.168336:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.168337:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.168339:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.168340:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.168341:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.168342:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.168344:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.168345:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.168347:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.168350:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.168355:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.168356:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.168357:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.168358:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.168359:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.168361:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.168367:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1776us (1917us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.168372:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57694 00000100:00000040:5.0:1496184950.168373:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.168375:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.168376:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.168378:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026072050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.168384:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.168384:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026072050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.168386:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.168387:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.168388:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.168390:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.168391:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.168392:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.168392:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.168393:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026072050. 00002000:00000001:5.0:1496184950.168394:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.168396:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.168397:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7e00. 00000020:00000010:5.0:1496184950.168399:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.168400:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.168402:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.168403:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.168405:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.168411:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.168412:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.196701:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.196703:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.196705:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdd850 x1568861104107184 msgsize 520 00000100:00100000:6.0:1496184950.196710:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.196711:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.196713:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.196760:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.196764:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.196766:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.196768:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.196769:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.196771:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.196774:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.196777:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107184 00000020:00000001:5.0:1496184950.196779:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.196781:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.196782:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.196784:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.196787:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.196789:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.196792:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.196794:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.196796:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.196810:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7c80. 00000020:00000010:5.0:1496184950.196811:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.196813:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.196813:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.196815:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.196817:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.196820:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdd850 x1568861104107184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.196825:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.196826:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.196830:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.196832:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.196833:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.196836:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43424 00000100:00000040:5.0:1496184950.196837:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.196838:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683671120 : -131872025880496 : ffff881026bdd850) 00000100:00000040:5.0:1496184950.196841:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdd850 x1568861104107184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.196846:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.196847:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.196848:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107184:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.196850:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107184 00000020:00000001:5.0:1496184950.196852:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.196854:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.196857:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.196859:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.196860:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.196862:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.196863:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.196864:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.196867:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.196868:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.196870:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.196872:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.196873:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.196875:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.196878:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.196879:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.196880:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.196881:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.196882:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.196883:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.196884:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.196885:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.196887:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.196888:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.196889:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.196902:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.196903:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.196906:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.196907:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.196908:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.196910:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.196911:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.196913:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.196915:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.196915:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.196916:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.196918:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.196920:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.196930:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.196931:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.196932:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.196934:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.196935:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.196936:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.196937:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.196939:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.196940:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.196941:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.196942:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.196943:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.196944:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.196946:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.196947:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.196949:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.196952:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.196953:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.196955:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.196956:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.196957:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.196958:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.196960:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.196961:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.196962:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.196963:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.196965:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.196966:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.196967:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.196967:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.196968:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.196970:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.196971:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.196972:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.196974:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.196975:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.196976:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.196977:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.196979:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.196980:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.196982:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.196982:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.196983:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.196984:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.196985:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.196990:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.196992:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.196993:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107184 00010000:00000001:5.0:1496184950.196995:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.196996:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdd850 x1568861104107184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.197002:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.197002:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.197004:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.197006:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.197007:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.197009:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.197010:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.197011:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.197012:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197013:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.197015:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.197017:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107184, offset 0 00000100:00000001:5.0:1496184950.197027:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197028:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.197030:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.197031:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.197032:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.197033:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197034:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.197036:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdd850 x1568861104107184/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.197041:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107184:12345-10.37.248.124@o2ib1:8 Request procesed in 194us (334us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.197046:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43424 00000100:00000040:5.0:1496184950.197047:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.197048:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.197049:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.197051:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7c80. 00000020:00000010:5.0:1496184950.197052:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.197054:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.197055:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.197057:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.197059:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.197062:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.197063:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.197116:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.197118:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.197119:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68a800. 02000000:00000001:7.0:1496184950.197122:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.197123:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.197560:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.197562:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.197564:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.197569:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.197570:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.197572:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.197601:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.197604:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.197606:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.197607:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.197608:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.197609:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.197611:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.197613:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.197615:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.197615:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.197617:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.197618:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.197619:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.197621:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.197623:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.197624:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.197626:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.197628:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7600. 00000020:00000010:5.0:1496184950.197630:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.197632:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.197633:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.197634:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.197635:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.197636:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.197637:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.197639:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.197640:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.197641:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.197642:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.197643:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.197644:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.197645:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.197646:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.197647:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.197647:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197648:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.197650:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197651:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197652:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.197653:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.197654:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.197655:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.197658:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.197664:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.197665:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.197667:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.197668:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.197668:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.197670:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.197672:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.197672:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.197673:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.197674:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00002000:00000001:5.0:1496184950.197675:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197676:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197678:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197683:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.197685:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.197686:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.197687:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.197687:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.197688:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.197689:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.197691:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57695 00000100:00000040:5.0:1496184950.197693:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.197694:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671693392 : -131872037858224 : ffff881026071450) 00000100:00000040:5.0:1496184950.197696:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.197702:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.197702:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.197704:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.197706:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.197707:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.197709:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.197710:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197711:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.197711:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.197713:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.197714:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.197715:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.197716:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.197717:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197718:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.197720:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.197721:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.197722:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.197723:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197725:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197726:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.197727:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.197729:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.197733:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.197735:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.197736:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.197738:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.197740:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.197741:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.197742:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.197744:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.197745:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.197746:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.197748:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.197750:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.197750:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.197751:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.197752:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.197754:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.197754:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.197755:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.197758:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.197779:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.197785:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197786:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.197789:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.197790:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.197791:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.197792:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.197793:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.197795:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.197796:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.197798:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.197802:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.197803:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.197829:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfeec0-0x592def0cfeec0 00000100:00000001:5.0:1496184950.197832:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.199496:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.199499:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.210616:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.210621:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.210622:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.221613:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.221670:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.221689:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.221691:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.221695:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.221696:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.221698:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.221700:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.221733:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.221735:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.221737:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.221738:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.221739:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.221740:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.221741:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.221742:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.221743:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.221744:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.221745:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.221747:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.221748:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.221750:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.221751:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.221752:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.221753:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.221755:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.221757:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.221759:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.221765:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.221766:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.221767:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.221767:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.221769:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.221770:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.221776:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24073us (24210us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.221781:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57695 00000100:00000040:5.0:1496184950.221783:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.221784:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.221785:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.221788:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.221793:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.221794:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.221795:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.221796:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.221798:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.221799:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.221801:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.221801:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.221802:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.221803:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071450. 00002000:00000001:5.0:1496184950.221804:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.221805:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.221806:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7600. 00000020:00000010:5.0:1496184950.221808:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.221810:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.221811:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.221813:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.221815:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.221820:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.221821:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.232691:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.232693:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.232695:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bddc50 x1568861104107216 msgsize 520 00000100:00100000:6.0:1496184950.232700:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.232701:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.232703:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.232750:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.232754:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.232757:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.232758:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.232760:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.232762:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.232765:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.232768:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107216 00000020:00000001:5.0:1496184950.232770:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.232772:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.232773:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.232775:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.232777:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.232780:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.232783:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.232784:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.232786:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.232800:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7080. 00000020:00000010:5.0:1496184950.232802:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.232803:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.232804:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.232805:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.232807:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.232810:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bddc50 x1568861104107216/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.232819:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.232820:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.232825:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.232827:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.232828:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.232828:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.232829:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.232830:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.232831:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.232833:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43425 00000100:00000040:5.0:1496184950.232835:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.232836:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683672144 : -131872025879472 : ffff881026bddc50) 00000100:00000040:5.0:1496184950.232838:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bddc50 x1568861104107216/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.232843:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.232844:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.232846:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107216:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.232848:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107216 00000020:00000001:5.0:1496184950.232849:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.232851:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.232854:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.232857:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.232858:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.232859:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.232860:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.232861:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.232864:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.232865:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.232867:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.232869:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.232870:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.232872:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.232875:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.232876:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.232877:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.232878:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.232879:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.232880:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.232881:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.232882:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.232884:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.232885:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.232886:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.232887:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.232888:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.232907:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.232908:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.232909:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.232911:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.232912:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.232914:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.232915:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.232916:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.232917:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.232919:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.232920:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.232922:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.232923:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.232924:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.232925:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.232927:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.232928:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.232929:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.232931:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.232932:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.232933:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.232934:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.232935:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.232937:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.232938:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.232939:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.232942:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.232944:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.232945:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.232947:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.232948:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.232949:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.232950:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.232951:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.232953:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.232954:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.232955:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.232957:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.232958:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.232959:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.232960:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.232961:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.232962:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.232963:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.232964:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.232966:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.232967:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.232968:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.232969:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.232971:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.232972:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.232974:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.232974:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.232975:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.232976:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.232977:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.232978:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.232979:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.232981:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107216 00010000:00000001:5.0:1496184950.232982:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.232984:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bddc50 x1568861104107216/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.232989:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.232990:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.232991:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.232993:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.232995:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.232997:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.232998:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.232999:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.233000:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233001:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.233003:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.233005:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107216, offset 0 00000100:00000001:5.0:1496184950.233015:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233016:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.233018:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.233019:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.233020:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.233020:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233021:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.233023:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bddc50 x1568861104107216/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.233029:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107216:12345-10.37.248.124@o2ib1:8 Request procesed in 184us (332us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.233033:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43425 00000100:00000040:5.0:1496184950.233035:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.233036:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.233037:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.233039:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7080. 00000020:00000010:5.0:1496184950.233040:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.233042:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.233043:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.233044:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.233046:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.233050:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.233050:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.233129:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.233132:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.233134:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68d000. 02000000:00000001:7.0:1496184950.233139:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.233140:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.233525:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.233527:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.233530:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.233535:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.233536:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.233538:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.233567:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.233570:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.233571:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.233572:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.233573:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.233574:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.233577:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.233578:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.233580:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.233581:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.233582:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.233583:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.233585:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.233586:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.233589:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.233589:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.233591:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.233593:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7700. 00000020:00000010:5.0:1496184950.233594:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.233596:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.233598:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.233599:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.233600:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.233601:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.233602:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.233604:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.233605:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.233606:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.233607:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.233608:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.233609:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.233610:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.233611:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.233612:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.233612:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233613:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.233615:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233616:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233617:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.233618:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.233619:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.233620:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.233623:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.233630:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.233630:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.233632:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.233633:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.233634:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.233635:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.233637:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.233638:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.233638:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.233639:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 00002000:00000001:5.0:1496184950.233640:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233642:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233644:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233648:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.233651:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.233651:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.233652:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.233653:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.233654:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.233655:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.233657:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57696 00000100:00000040:5.0:1496184950.233659:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.233660:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671272528 : -131872038279088 : ffff88102600a850) 00000100:00000040:5.0:1496184950.233662:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.233667:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.233668:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.233670:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.233672:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.233673:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.233675:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.233676:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233677:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.233677:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.233679:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.233680:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.233681:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.233682:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.233682:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233683:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.233685:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.233686:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.233687:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.233689:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233690:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233691:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.233692:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.233694:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.233698:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.233701:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.233701:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.233704:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.233705:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.233706:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.233707:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.233709:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.233710:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.233711:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.233713:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.233715:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.233715:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.233716:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.233717:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.233719:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.233719:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.233720:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.233723:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.233743:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.233748:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233750:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.233752:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.233753:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.233755:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.233755:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.233757:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.233758:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184950.233759:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.233761:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.233765:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.233766:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.233792:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfeee0-0x592def0cfeee0 00000100:00000001:5.0:1496184950.233794:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.235374:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.235388:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.246433:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.246439:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.246439:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.257468:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:1.0:1496184950.257505:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184950.257508:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184950.257511:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b8ac50 x1568861104107248 msgsize 520 00000100:00100000:1.0:1496184950.257518:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184950.257520:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184950.257523:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.257529:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00010000:00020000:5.0:1496184950.257532:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00004000:00000001:3.0:1496184950.257532:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.257536:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184950.257547:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.257549:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00010000:00000001:5.0:1496184950.257551:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184950.257551:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.257554:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 02000000:00000001:3.0:1496184950.257555:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.257557:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.257558:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00100000:3.0:1496184950.257559:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107248 00002000:00000001:5.0:1496184950.257561:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.257562:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000001:3.0:1496184950.257562:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184950.257563:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184950.257565:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184950.257568:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184950.257570:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184950.257571:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184950.257576:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.257577:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184950.257582:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000010:3.0:1496184950.257586:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8700. 00000020:00000010:3.0:1496184950.257589:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184950.257592:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184950.257593:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184950.257594:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000001:00000001:5.0:1496184950.257596:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.257598:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000100:00000001:3.0:1496184950.257599:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00002000:00000001:5.0:1496184950.257600:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.257601:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.257602:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.257603:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.257604:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000200:3.0:1496184950.257605:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b8ac50 x1568861104107248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000001:5.0:1496184950.257606:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.257606:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.257607:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.257608:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000010:5.0:1496184950.257610:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000001:3.0:1496184950.257610:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184950.257611:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000010:5.0:1496184950.257612:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.257613:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.257614:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.257615:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b400. 00000100:00000001:3.0:1496184950.257615:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 02000000:00000001:5.0:1496184950.257616:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.257618:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00004000:00000001:3.0:1496184950.257619:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000020:00000001:5.0:1496184950.257620:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00004000:00000001:3.0:1496184950.257620:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184950.257621:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.257622:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00010000:00000080:5.0:1496184950.257623:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.257623:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184950.257624:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000040:5.0:1496184950.257628:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00100000:3.0:1496184950.257628:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6110 00010000:00000001:5.0:1496184950.257629:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00000100:00000040:3.0:1496184950.257630:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00010000:00000001:5.0:1496184950.257631:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.257631:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257632:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736391248 : -131906973160368 : ffff880803b8ac50) 00000020:00000001:5.0:1496184950.257633:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.257635:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184950.257635:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b8ac50 x1568861104107248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184950.257640:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.257641:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.257642:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23972us (24110us total) trans 0 rc -110/-999 00000100:00100000:3.0:1496184950.257643:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104107248:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184950.257646:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107248 00000100:00100000:5.0:1496184950.257647:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57696 00000020:00000001:3.0:1496184950.257648:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000100:00000040:5.0:1496184950.257649:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.257650:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.257651:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00010000:00000001:3.0:1496184950.257652:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00002000:00100000:5.0:1496184950.257654:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.257660:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.257660:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a850 with x1568861102817424 ext(0->1048575) 00000020:00000040:3.0:1496184950.257660:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00010000:00000001:5.0:1496184950.257662:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.257663:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.257664:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000100:00000001:3.0:1496184950.257665:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 00010000:00000001:5.0:1496184950.257666:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 02000000:00000001:3.0:1496184950.257666:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 00010000:00000001:5.0:1496184950.257668:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.257668:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.257669:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.257669:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a850. 02000000:00000010:3.0:1496184950.257669:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fec00. 00002000:00000001:5.0:1496184950.257671:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184950.257671:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.257672:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:3.0:1496184950.257673:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:5.0:1496184950.257674:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7700. 00000020:00000010:5.0:1496184950.257675:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.257677:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.257678:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.257679:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000001:5.0:1496184950.257680:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.257682:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00010000:00000001:3.0:1496184950.257682:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184950.257685:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00004000:00000001:5.0:1496184950.257687:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00010000:00000001:3.0:1496184950.257687:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00004000:00000001:5.0:1496184950.257689:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00000001:3.0:1496184950.257689:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184950.257692:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:3.0:1496184950.257695:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.257698:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.257701:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.257703:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.257705:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.257706:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.257708:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.257710:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.257712:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.257713:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.257715:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.257716:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.257718:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.257730:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.257732:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.257733:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.257736:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.257737:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.257740:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.257742:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.257743:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.257744:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.257746:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.257749:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.257751:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.257752:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.257754:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.257756:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.257757:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.257759:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.257760:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.257762:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.257764:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.257765:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.257766:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.257768:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.257770:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.257771:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257773:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.257777:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.257779:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.257782:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.257784:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.257786:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.257787:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257788:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.257789:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.257791:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.257793:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.257795:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.257797:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257799:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.257801:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.257803:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.257804:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.257806:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.257808:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.257809:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.257812:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.257813:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.257815:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.257817:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.257820:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.257822:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.257823:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.257824:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.257825:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.257827:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.257828:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.257830:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.257832:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.257834:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107248 00010000:00000001:3.0:1496184950.257836:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.257839:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b8ac50 x1568861104107248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.257846:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.257847:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.257849:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.257852:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.257854:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.257855:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.257856:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.257858:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.257859:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257861:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.257863:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.257867:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107248, offset 0 00000100:00000001:3.0:1496184950.257884:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.257885:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.257886:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.257888:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.257903:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.257905:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.257906:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.257908:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b8ac50 x1568861104107248/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.257914:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107248:12345-10.37.248.124@o2ib1:8 Request procesed in 272us (401us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.257919:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6110 00000100:00000040:3.0:1496184950.257921:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.257923:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.257924:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.257927:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8700. 00000020:00000010:3.0:1496184950.257929:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184950.257931:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000040:3.0:1496184950.257933:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.257935:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.257938:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.257945:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.257946:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.258310:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.258319:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.258321:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fec00. 02000000:00000001:7.0:1496184950.258323:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.258324:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.258673:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.258675:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.258678:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026071850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.258683:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.258684:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.258686:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.258716:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.258718:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.258720:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.258721:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.258723:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.258724:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.258726:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.258728:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.258730:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.258731:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.258732:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.258733:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.258735:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.258736:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.258739:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.258740:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.258741:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.258743:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7300. 00000020:00000010:5.0:1496184950.258745:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.258747:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.258749:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.258750:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.258751:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.258752:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.258753:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.258754:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.258756:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.258757:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.258758:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.258759:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.258760:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.258761:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.258762:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.258763:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.258763:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258764:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.258766:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258767:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258768:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.258770:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.258771:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.258772:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.258775:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.258781:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.258782:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.258784:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.258784:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.258785:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.258787:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.258788:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.258789:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.258789:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.258790:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00002000:00000001:5.0:1496184950.258791:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.258792:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.258795:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.258799:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.258801:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.258810:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.258811:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.258811:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.258812:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.258813:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.258816:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57697 00000100:00000040:5.0:1496184950.258817:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.258819:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671694416 : -131872037857200 : ffff881026071850) 00000100:00000040:5.0:1496184950.258821:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.258826:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.258827:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.258829:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.258831:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.258832:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.258834:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.258835:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258836:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.258837:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.258838:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.258839:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.258840:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.258841:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.258842:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258843:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.258845:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.258846:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.258847:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.258848:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.258850:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258851:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.258852:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.258854:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.258858:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.258860:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.258861:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.258863:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.258865:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.258867:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.258867:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.258869:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.258871:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.258872:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.258874:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.258876:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.258876:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.258877:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.258878:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.258880:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.258880:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.258881:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.258884:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.258917:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.258924:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258925:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.258928:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.258929:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.258930:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.258931:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.258932:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.258934:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184950.258935:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.258937:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.258941:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.258942:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.258969:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfef00-0x592def0cfef00 00000100:00000001:5.0:1496184950.258971:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.269906:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.269908:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:6.0:1496184950.274946:0:13696:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184950.274948:0:13696:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184950.274952:0:13696:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184950.274953:0:13696:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.274954:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184950.274956:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.274957:0:13696:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184950.274958:0:13696:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184950.277958:0:13694:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:5.0:1496184950.277959:0:13695:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:7.0:1496184950.277960:0:13694:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.277960:0:13695:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:7.0:1496184950.277964:0:13694:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:5.0:1496184950.277964:0:13695:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:7.0:1496184950.277965:0:13694:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.277965:0:13695:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184950.277966:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:5.0:1496184950.277967:0:13695:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184950.277968:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.277968:0:13695:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:7.0:1496184950.277969:0:13694:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:7.0:1496184950.277970:0:13694:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.277970:0:13695:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:5.0:1496184950.277971:0:13695:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.280896:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.280901:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.280902:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00010000:00000001:6.0:1496184950.280945:0:13722:0:(ldlm_pool.c:339:ldlm_srv_pool_recalc()) Process entered 00010000:00000001:6.0:1496184950.280947:0:13722:0:(ldlm_pool.c:371:ldlm_srv_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:6.0:1496184950.280952:0:13722:0:(ldlm_pool.c:478:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:6.0:1496184950.280953:0:13722:0:(ldlm_pool.c:482:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184950.280987:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184950.280989:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:2.0:1496184950.280991:0:13720:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1496184950.280992:0:13720:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184950.281007:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184950.281011:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184950.281013:0:19518:0:(genops.c:1785:obd_stale_export_get()) Process entered 00000020:00000001:1.0:1496184950.281014:0:19518:0:(genops.c:1799:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.290948:0:13693:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:6.0:1496184950.290950:0:13693:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:6.0:1496184950.290954:0:13693:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:6.0:1496184950.290955:0:13693:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.290956:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184950.290958:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.290959:0:13693:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:6.0:1496184950.290960:0:13693:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.291873:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.291933:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.291952:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.291954:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.291958:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.291959:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.291961:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.291963:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.291996:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.291998:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.292000:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.292001:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.292002:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.292003:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.292004:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.292006:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.292007:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.292007:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.292009:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.292010:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.292012:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.292013:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.292014:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.292015:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.292016:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.292018:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.292020:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.292022:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.292028:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.292029:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.292030:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.292031:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.292032:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.292034:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.292040:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33212us (33360us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.292045:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57697 00000100:00000040:5.0:1496184950.292047:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.292048:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.292049:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.292052:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.292057:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.292058:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.292060:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.292060:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.292062:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.292063:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.292065:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.292065:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.292066:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.292066:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071850. 00002000:00000001:5.0:1496184950.292068:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.292069:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.292070:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7300. 00000020:00000010:5.0:1496184950.292072:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.292074:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.292075:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.292077:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.292079:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.292084:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.292085:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:1.0:1496184950.302975:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:1.0:1496184950.302977:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:1.0:1496184950.302980:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88850 x1568861104107280 msgsize 520 00000100:00100000:1.0:1496184950.302986:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184950.302988:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184950.302990:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.303024:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.303027:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.303029:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184950.303031:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.303033:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184950.303034:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184950.303038:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184950.303041:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107280 00000020:00000001:3.0:1496184950.303043:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:3.0:1496184950.303047:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:3.0:1496184950.303048:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:3.0:1496184950.303050:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:3.0:1496184950.303052:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184950.303054:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184950.303058:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.303059:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184950.303060:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000010:3.0:1496184950.303063:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8800. 00000020:00000010:3.0:1496184950.303065:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184950.303067:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184950.303068:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184950.303069:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184950.303073:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:3.0:1496184950.303077:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b88850 x1568861104107280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:3.0:1496184950.303083:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:3.0:1496184950.303084:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303089:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:3.0:1496184950.303092:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.303093:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184950.303094:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:2.0:1496184950.303094:0:14310:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.303095:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.303096:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:3.0:1496184950.303097:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00004000:00000001:2.0:1496184950.303097:0:14310:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184950.303099:0:14310:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00100000:3.0:1496184950.303100:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6111 00000100:00000001:2.0:1496184950.303100:0:14310:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000040:3.0:1496184950.303102:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:2.0:1496184950.303102:0:14310:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303103:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736382032 : -131906973169584 : ffff880803b88850) 00000100:00000001:2.0:1496184950.303104:0:14310:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.303106:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88850 x1568861104107280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00004000:00000001:2.0:1496184950.303107:0:14310:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184950.303108:0:14310:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184950.303111:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.303112:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184950.303114:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107280:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184950.303118:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107280 00000020:00000001:3.0:1496184950.303120:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184950.303124:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184950.303129:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184950.303134:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184950.303135:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184950.303137:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23ff000. 02000000:00000001:3.0:1496184950.303138:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303140:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.303143:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184950.303145:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184950.303148:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184950.303150:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184950.303152:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184950.303154:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:3.0:1496184950.303157:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.303159:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.303160:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.303162:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.303163:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.303164:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.303165:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.303167:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.303169:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.303170:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.303171:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.303172:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.303173:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.303177:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.303178:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.303179:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.303183:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.303184:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.303187:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.303188:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.303189:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.303190:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.303192:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.303194:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.303195:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.303196:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.303198:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.303200:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.303201:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.303202:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.303203:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.303205:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.303206:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.303207:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.303209:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.303210:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.303212:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.303213:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303214:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.303218:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.303220:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.303222:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.303224:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.303225:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.303226:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303227:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.303229:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.303231:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.303233:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.303235:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.303236:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303237:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.303239:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.303240:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.303241:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.303243:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.303244:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.303245:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.303247:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.303248:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.303249:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.303251:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.303253:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.303255:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.303256:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.303257:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.303258:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.303259:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.303260:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.303262:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.303263:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.303265:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107280 00010000:00000001:3.0:1496184950.303267:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.303269:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b88850 x1568861104107280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.303275:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.303276:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.303277:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.303279:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.303281:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.303283:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.303284:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.303286:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.303287:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303288:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.303290:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.303293:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107280, offset 0 00000100:00000001:3.0:1496184950.303311:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.303312:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.303313:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.303314:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.303316:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.303317:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.303318:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.303320:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88850 x1568861104107280/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.303326:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107280:12345-10.37.248.124@o2ib1:8 Request procesed in 213us (344us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.303330:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6111 00000100:00000040:3.0:1496184950.303332:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.303334:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.303335:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.303337:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8800. 00000020:00000010:3.0:1496184950.303339:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184950.303340:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000040:3.0:1496184950.303342:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.303343:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.303345:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.303349:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.303350:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.303373:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.303375:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.303377:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23ff000. 02000000:00000001:7.0:1496184950.303380:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.303381:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.303883:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.303885:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.303887:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070c50 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.303903:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.303904:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.303906:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.303937:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.303940:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.303941:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.303943:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.303944:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.303945:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.303947:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.303950:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.303951:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.303952:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.303954:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.303955:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.303957:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.303958:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.303961:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.303962:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.303964:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.303966:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7500. 00000020:00000010:5.0:1496184950.303968:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.303970:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.303971:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.303972:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.303973:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.303974:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.303975:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.303977:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.303978:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.303979:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.303980:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.303982:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.303982:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.303984:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.303984:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.303985:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.303986:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.303987:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.303988:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.303989:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.303990:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.303992:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.303993:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.303994:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.303997:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.304003:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.304004:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.304006:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.304006:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.304007:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.304009:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.304010:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.304011:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.304011:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.304012:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00002000:00000001:5.0:1496184950.304013:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.304014:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.304017:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.304021:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.304023:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.304024:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.304025:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.304026:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.304027:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.304028:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.304030:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57698 00000100:00000040:5.0:1496184950.304032:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.304033:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671691344 : -131872037860272 : ffff881026070c50) 00000100:00000040:5.0:1496184950.304036:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.304041:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.304041:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.304043:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.304046:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.304047:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.304049:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.304050:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.304051:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.304051:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.304053:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.304054:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.304055:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.304056:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.304057:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.304058:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.304060:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.304061:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.304062:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.304063:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.304065:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.304066:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.304067:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.304069:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.304073:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.304076:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.304076:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.304079:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.304080:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.304082:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.304085:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.304086:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.304088:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.304089:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.304091:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.304093:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.304093:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.304094:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.304095:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.304097:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.304098:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.304099:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.304101:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.304123:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.304128:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.304130:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.304132:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.304134:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.304135:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.304136:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.304137:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.304138:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb5000. 00000100:00000010:5.0:1496184950.304140:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.304141:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.304146:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.304147:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.304173:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfef20-0x592def0cfef20 00000100:00000001:5.0:1496184950.304175:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.305820:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.305823:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.305843:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.305848:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.305851:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb5000 00000100:00000001:3.0:1496184950.305856:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.305905:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.305920:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.305922:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.305926:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.305927:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.305929:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.305930:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.305964:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.305966:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.305968:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.305969:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.305969:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.305971:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.305972:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.305973:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.305974:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.305975:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.305977:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.305978:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.305980:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb5000. 00000100:00000001:5.0:1496184950.305981:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.305982:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.305983:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.305985:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.305986:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.305988:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.305991:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.305996:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.305997:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.305998:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.305999:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.306000:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.306002:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.306008:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1966us (2119us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.306013:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57698 00000100:00000040:5.0:1496184950.306015:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.306017:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.306017:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.306020:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.306025:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.306026:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.306028:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.306029:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.306030:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.306032:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.306033:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.306034:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.306035:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.306035:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070c50. 00002000:00000001:5.0:1496184950.306037:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.306038:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.306039:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7500. 00000020:00000010:5.0:1496184950.306041:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.306043:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.306044:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.306046:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.306048:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.306053:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.306054:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.335020:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.335022:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.335024:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde450 x1568861104107312 msgsize 520 00000100:00100000:6.0:1496184950.335029:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.335031:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.335033:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.335063:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.335066:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.335067:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.335068:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.335070:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.335071:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.335073:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.335075:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107312 00000020:00000001:5.0:1496184950.335076:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.335077:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.335078:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.335080:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.335081:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.335083:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.335085:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.335086:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.335087:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.335089:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7580. 00000020:00000010:5.0:1496184950.335091:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.335092:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.335093:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.335094:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.335096:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.335099:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde450 x1568861104107312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.335105:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.335105:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335110:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.335112:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.335113:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.335114:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.335115:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.335115:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.335116:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.335118:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43426 00000100:00000040:5.0:1496184950.335120:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.335121:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683674192 : -131872025877424 : ffff881026bde450) 00000100:00000040:5.0:1496184950.335123:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde450 x1568861104107312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.335129:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.335129:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.335131:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107312:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.335133:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107312 00000020:00000001:5.0:1496184950.335134:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.335136:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.335139:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.335142:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.335143:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.335144:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.335146:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335147:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.335149:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.335151:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.335153:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.335155:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.335156:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.335158:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.335161:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.335162:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.335164:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.335165:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.335166:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.335167:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.335168:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.335170:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.335171:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.335172:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.335173:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.335174:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.335176:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.335179:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.335180:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.335181:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.335183:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.335184:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.335186:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.335187:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.335188:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.335189:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.335191:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.335193:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.335194:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.335196:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.335197:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.335199:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.335200:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.335201:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.335202:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.335204:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.335205:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.335206:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.335207:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.335208:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.335210:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.335211:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335213:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.335215:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.335218:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.335219:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.335221:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.335222:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.335223:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335224:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.335226:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.335227:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.335229:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.335230:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.335232:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335233:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.335234:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.335235:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.335236:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.335238:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.335238:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.335239:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.335242:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.335243:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.335244:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.335245:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335247:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.335248:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.335249:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.335250:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.335251:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.335252:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.335253:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335254:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.335255:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.335257:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107312 00010000:00000001:5.0:1496184950.335258:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.335260:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde450 x1568861104107312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.335265:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.335266:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.335267:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.335270:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.335271:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.335273:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.335274:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.335275:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.335276:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335277:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.335279:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.335281:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107312, offset 0 00000100:00000001:5.0:1496184950.335292:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335293:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.335294:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.335295:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.335297:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.335297:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335298:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.335300:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde450 x1568861104107312/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.335306:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107312:12345-10.37.248.124@o2ib1:8 Request procesed in 176us (280us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.335310:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43426 00000100:00000040:5.0:1496184950.335312:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.335313:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.335314:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.335315:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7580. 00000020:00000010:5.0:1496184950.335317:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.335318:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.335320:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.335321:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.335323:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.335326:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.335327:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.335379:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.335381:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.335382:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68b400. 02000000:00000001:6.0:1496184950.335385:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.335386:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.335808:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.335810:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.335812:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026071050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.335817:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.335818:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.335820:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.335852:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.335855:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.335856:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.335857:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.335858:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.335859:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.335862:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.335864:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.335865:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.335866:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.335867:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.335868:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.335870:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.335871:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.335874:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.335875:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.335876:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.335878:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7f80. 00000020:00000010:5.0:1496184950.335880:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.335882:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.335884:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.335885:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.335886:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.335887:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.335888:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.335899:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.335901:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.335902:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.335903:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.335904:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.335905:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.335906:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.335907:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.335908:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.335908:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335909:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.335911:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335912:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335913:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.335914:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.335915:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.335916:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.335919:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.335926:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.335926:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.335928:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.335929:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.335930:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.335931:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.335933:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.335934:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.335934:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.335935:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00002000:00000001:5.0:1496184950.335936:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335937:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335940:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335944:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.335946:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.335947:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.335948:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.335948:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.335949:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.335950:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.335953:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57699 00000100:00000040:5.0:1496184950.335954:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.335955:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671692368 : -131872037859248 : ffff881026071050) 00000100:00000040:5.0:1496184950.335958:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.335963:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.335964:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.335965:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.335967:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.335968:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.335970:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.335971:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335972:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.335973:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.335975:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.335976:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.335977:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.335977:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.335978:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335979:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.335981:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.335982:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.335983:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.335985:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.335986:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.335987:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.335988:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.335990:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.335994:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.335997:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.335997:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.336000:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.336002:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.336002:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.336003:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.336005:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.336007:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.336007:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.336009:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.336011:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.336011:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.336012:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.336013:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.336015:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.336015:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.336016:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.336019:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.336043:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.336048:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.336050:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.336052:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.336054:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.336055:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.336055:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.336057:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.336058:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.336060:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.336061:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.336066:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.336067:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.336094:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfef40-0x592def0cfef40 00000100:00000001:5.0:1496184950.336096:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.337687:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.337701:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.337707:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.337712:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.337713:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.337718:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.337754:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.337768:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.337770:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.337774:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.337775:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.337777:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.337779:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.337812:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.337814:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.337816:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.337817:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.337818:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.337819:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.337821:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.337822:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.337823:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.337823:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.337824:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.337826:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.337828:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.337829:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.337830:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.337831:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.337832:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.337834:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.337836:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.337839:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.337844:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.337845:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.337846:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.337846:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.337848:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.337849:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.337855:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1891us (2042us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.337860:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57699 00000100:00000040:5.0:1496184950.337862:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.337864:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.337864:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.337867:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026071050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.337873:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.337873:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026071050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.337875:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.337876:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.337877:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.337878:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.337880:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.337881:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.337881:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.337882:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026071050. 00002000:00000001:5.0:1496184950.337883:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.337884:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.337886:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7f80. 00000020:00000010:5.0:1496184950.337888:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.337897:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.337899:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.337901:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.337903:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.337909:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.337910:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:3.0:1496184950.341954:0:13691:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:3.0:1496184950.341957:0:13691:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184950.341962:0:13691:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184950.341963:0:13691:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.341965:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184950.341968:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.341969:0:13691:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184950.341970:0:13691:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.342911:0:13689:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184950.342914:0:13689:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1496184950.342920:0:13689:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1496184950.342922:0:13689:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.342924:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1496184950.342926:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.342928:0:13689:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184950.342928:0:13690:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:1.0:1496184950.342930:0:13689:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184950.342931:0:13690:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184950.342937:0:13690:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184950.342938:0:13692:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:0.0:1496184950.342938:0:13690:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184950.342940:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184950.342941:0:13692:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1496184950.342943:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1496184950.342945:0:13690:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1496184950.342947:0:13690:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.342948:0:13692:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:3.0:1496184950.342949:0:13692:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.342951:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184950.342968:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.342970:0:13692:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:3.0:1496184950.342970:0:13692:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:6.0:1496184950.366685:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.366687:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.366689:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde050 x1568861104107344 msgsize 520 00000100:00100000:6.0:1496184950.366694:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.366695:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.366698:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.366729:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.366732:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.366734:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.366735:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.366736:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.366737:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.366740:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.366742:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107344 00000020:00000001:5.0:1496184950.366743:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.366744:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.366745:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.366746:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.366748:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.366750:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.366752:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.366752:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.366754:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.366756:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7800. 00000020:00000010:5.0:1496184950.366758:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.366759:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.366760:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.366761:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.366763:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.366767:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde050 x1568861104107344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.366772:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.366773:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366777:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.366780:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.366780:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.366781:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.366782:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.366783:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.366784:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.366786:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43427 00000100:00000040:5.0:1496184950.366788:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.366789:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683673168 : -131872025878448 : ffff881026bde050) 00000100:00000040:5.0:1496184950.366792:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde050 x1568861104107344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.366797:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.366797:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.366799:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107344:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.366801:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107344 00000020:00000001:5.0:1496184950.366802:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.366804:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.366807:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.366810:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.366811:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.366812:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.366813:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366815:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.366817:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.366818:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.366820:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.366822:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.366823:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.366825:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.366828:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.366829:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.366830:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.366831:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.366832:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.366833:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.366834:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.366835:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.366837:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.366838:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.366839:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.366840:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.366841:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.366844:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.366845:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.366846:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.366848:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.366849:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.366851:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.366852:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.366853:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.366854:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.366856:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.366858:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.366859:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.366860:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.366861:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.366863:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.366864:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.366865:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.366866:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.366867:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.366868:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.366869:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.366870:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.366872:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.366873:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.366875:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366876:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.366878:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.366881:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.366882:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.366884:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.366885:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.366886:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366887:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.366888:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.366900:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.366902:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.366903:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.366904:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366905:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.366906:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.366907:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.366908:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.366910:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.366911:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.366912:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.366913:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.366915:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.366916:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.366917:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.366919:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.366920:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.366921:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.366922:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.366923:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.366924:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.366925:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.366926:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.366927:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.366929:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107344 00010000:00000001:5.0:1496184950.366930:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.366932:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde050 x1568861104107344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.366937:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.366938:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.366939:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.366941:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.366943:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.366945:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.366946:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.366947:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.366948:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366949:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.366951:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.366953:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107344, offset 0 00000100:00000001:5.0:1496184950.366964:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.366965:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.366966:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.366967:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.366968:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.366969:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.366970:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.366972:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde050 x1568861104107344/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.366977:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107344:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (286us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.366982:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43427 00000100:00000040:5.0:1496184950.366983:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.366985:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.366985:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.366987:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7800. 00000020:00000010:5.0:1496184950.366989:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.366990:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.366992:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.366993:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.366995:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.367000:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.367001:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.367055:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.367057:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.367058:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68a400. 02000000:00000001:6.0:1496184950.367061:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.367062:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.367487:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.367489:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.367492:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070850 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.367496:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.367498:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.367500:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.367533:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.367536:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.367537:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.367539:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.367540:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.367541:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.367543:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.367545:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.367546:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.367547:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.367548:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.367549:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.367551:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.367553:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.367555:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.367555:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.367557:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.367559:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7d00. 00000020:00000010:5.0:1496184950.367561:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.367563:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.367564:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.367565:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.367566:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.367567:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.367568:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.367569:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.367571:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.367572:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.367573:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.367574:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.367575:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.367576:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.367577:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.367578:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.367578:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367579:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.367581:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367582:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367583:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.367584:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.367585:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.367586:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.367589:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.367595:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.367596:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.367598:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.367599:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.367600:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.367601:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.367603:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.367603:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.367604:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.367604:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00002000:00000001:5.0:1496184950.367606:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.367607:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.367609:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.367613:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.367615:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.367616:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.367617:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.367618:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.367619:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.367620:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.367623:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57700 00000100:00000040:5.0:1496184950.367624:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.367625:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671690320 : -131872037861296 : ffff881026070850) 00000100:00000040:5.0:1496184950.367628:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.367633:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.367634:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.367635:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.367637:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.367638:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.367640:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.367641:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367642:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.367643:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.367644:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.367645:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.367646:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.367647:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.367647:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367649:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.367650:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.367651:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.367653:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.367654:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.367655:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367656:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.367657:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.367659:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.367664:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.367666:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.367667:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.367669:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.367671:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.367672:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.367673:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.367674:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.367676:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.367677:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.367679:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.367680:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.367681:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.367682:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.367683:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.367684:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.367685:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.367686:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.367688:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.367710:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.367716:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367717:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.367720:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.367721:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.367722:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.367723:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.367724:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.367725:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.367727:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.367728:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.367733:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.367734:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.367761:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfef60-0x592def0cfef60 00000100:00000001:5.0:1496184950.367763:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.369297:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.369312:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.369318:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.369322:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.369323:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.369328:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.369368:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.369384:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.369386:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.369390:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.369391:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.369393:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.369395:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.369428:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.369430:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.369432:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.369433:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.369434:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.369435:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.369436:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.369438:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.369438:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.369439:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.369440:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.369442:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.369443:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.369445:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.369446:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.369447:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.369448:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.369450:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.369452:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.369454:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.369459:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.369460:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.369461:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.369462:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.369463:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.369465:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.369471:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1837us (1978us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.369476:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57700 00000100:00000040:5.0:1496184950.369478:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.369479:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.369480:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.369483:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.369488:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.369489:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.369490:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.369491:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.369492:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.369494:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.369495:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.369496:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.369497:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.369497:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070850. 00002000:00000001:5.0:1496184950.369499:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.369500:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.369501:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7d00. 00000020:00000010:5.0:1496184950.369503:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.369505:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.369506:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.369508:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.369510:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.369515:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.369516:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.397859:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.397861:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.397863:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026bde850 x1568861104107376 msgsize 520 00000100:00100000:7.0:1496184950.397868:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.397869:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.397871:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.397905:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.397907:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.397909:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.397910:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.397911:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.397912:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.397915:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.397916:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107376 00000020:00000001:5.0:1496184950.397918:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.397919:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.397920:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.397921:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.397923:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.397924:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.397926:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.397927:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.397934:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.397936:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7680. 00000020:00000010:5.0:1496184950.397938:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.397939:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.397940:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.397941:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.397943:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.397946:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bde850 x1568861104107376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.397951:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.397952:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.397957:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.397959:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.397960:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.397960:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.397961:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.397962:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.397963:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.397965:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43428 00000100:00000040:5.0:1496184950.397967:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.397968:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683675216 : -131872025876400 : ffff881026bde850) 00000100:00000040:5.0:1496184950.397970:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bde850 x1568861104107376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.397976:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.397976:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.397978:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107376:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.397980:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107376 00000020:00000001:5.0:1496184950.397981:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.397983:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.397986:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.397989:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.397990:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.397991:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.397992:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.397994:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.397996:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.397998:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.398000:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.398001:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.398002:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.398005:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.398007:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.398008:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.398009:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.398010:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.398011:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.398012:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.398013:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.398014:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.398016:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.398017:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.398018:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.398019:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.398020:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.398024:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.398024:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.398026:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.398028:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.398028:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.398030:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.398032:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.398033:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.398034:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.398035:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.398037:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.398038:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398039:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.398040:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.398042:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.398043:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398044:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.398045:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.398047:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.398048:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.398049:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.398050:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.398051:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.398052:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.398054:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398055:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.398057:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.398060:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.398061:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.398063:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.398064:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.398065:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398067:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.398068:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.398069:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.398070:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.398072:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.398073:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398074:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.398075:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.398076:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.398077:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.398078:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.398079:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.398080:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.398082:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.398083:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.398084:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.398086:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398087:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.398088:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.398090:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.398090:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.398091:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.398092:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.398093:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398094:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.398095:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.398097:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107376 00010000:00000001:5.0:1496184950.398098:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.398100:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bde850 x1568861104107376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.398105:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.398107:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.398108:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.398110:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.398112:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.398113:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.398114:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.398115:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.398116:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398117:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.398119:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.398121:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107376, offset 0 00000100:00000001:5.0:1496184950.398132:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398133:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.398134:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.398135:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.398136:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.398137:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398138:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.398139:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bde850 x1568861104107376/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.398145:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107376:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (281us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.398149:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43428 00000100:00000040:5.0:1496184950.398151:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.398152:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.398162:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.398163:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7680. 00000020:00000010:5.0:1496184950.398165:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.398166:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.398168:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.398169:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.398171:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.398174:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.398175:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.398241:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.398244:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.398246:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68ac00. 02000000:00000001:6.0:1496184950.398251:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.398252:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.398676:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.398677:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.398680:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.398685:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.398686:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.398688:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.398717:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.398720:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.398722:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.398723:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.398724:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.398725:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.398728:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.398729:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.398731:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.398732:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.398733:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.398734:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.398736:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.398737:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.398740:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.398740:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.398742:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.398744:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7f00. 00000020:00000010:5.0:1496184950.398745:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.398747:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.398749:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.398750:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.398751:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.398752:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.398753:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.398754:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.398755:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.398757:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.398757:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.398759:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.398760:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.398761:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.398761:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.398762:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.398763:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398764:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.398766:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398767:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398768:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398769:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.398770:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398771:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.398774:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.398780:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.398781:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.398783:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.398784:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.398785:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.398786:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.398788:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.398788:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.398789:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.398790:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00002000:00000001:5.0:1496184950.398791:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398792:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398794:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398799:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.398801:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.398802:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.398803:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.398803:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.398804:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.398805:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.398808:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57701 00000100:00000040:5.0:1496184950.398809:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.398810:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671688272 : -131872037863344 : ffff881026070050) 00000100:00000040:5.0:1496184950.398812:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.398818:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.398819:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.398820:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.398823:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.398824:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.398825:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.398826:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398827:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.398828:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.398830:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.398831:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.398832:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.398832:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.398833:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398834:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.398836:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.398837:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.398838:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.398839:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398841:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398842:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.398843:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.398845:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.398849:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.398851:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.398852:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.398854:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.398856:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.398857:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.398858:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.398860:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.398862:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.398862:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.398864:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.398866:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.398867:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.398867:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.398868:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.398870:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.398871:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.398872:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.398874:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.398908:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.398914:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398915:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.398918:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398919:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.398920:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.398921:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.398923:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.398932:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.398933:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.398935:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.398939:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.398941:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.398966:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfef80-0x592def0cfef80 00000100:00000001:5.0:1496184950.398969:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.400655:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.400669:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.411779:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.411783:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.411784:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.422772:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.422850:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.422856:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.422859:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.422863:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.422863:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.422866:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.422867:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.422913:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.422915:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.422917:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.422918:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.422919:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.422921:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.422922:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.422923:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.422924:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.422925:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.422926:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.422927:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.422929:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.422930:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.422931:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.422932:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.422934:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.422936:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.422937:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.422940:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.422945:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.422947:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.422947:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.422948:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.422950:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.422951:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.422957:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 24138us (24276us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.422962:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57701 00000100:00000040:5.0:1496184950.422964:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.422965:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.422966:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.422969:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.422974:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.422975:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.422977:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.422977:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.422979:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.422980:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.422982:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.422982:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.422983:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.422983:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070050. 00002000:00000001:5.0:1496184950.422985:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.422986:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.422987:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7f00. 00000020:00000010:5.0:1496184950.422989:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.422991:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.422992:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.422994:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.422996:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.423001:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.423002:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.432876:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.432878:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.432881:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdec50 x1568861104107408 msgsize 520 00000100:00100000:6.0:1496184950.432886:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.432887:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.432889:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.432921:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.432924:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.432925:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.432926:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.432927:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.432928:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.432931:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.432933:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107408 00000020:00000001:5.0:1496184950.432934:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.432935:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.432936:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.432937:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.432939:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.432940:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.432943:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.432944:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.432945:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.432947:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7c00. 00000020:00000010:5.0:1496184950.432949:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.432950:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.432951:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.432952:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.432954:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.432957:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdec50 x1568861104107408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.432962:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.432963:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.432968:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.432970:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.432971:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.432972:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.432972:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.432973:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.432974:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.432977:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43429 00000100:00000040:5.0:1496184950.432978:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.432979:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683676240 : -131872025875376 : ffff881026bdec50) 00000100:00000040:5.0:1496184950.432982:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdec50 x1568861104107408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.432987:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.432988:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.432989:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107408:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.432991:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107408 00000020:00000001:5.0:1496184950.432993:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.432995:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.432998:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.433000:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.433001:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.433003:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68ac00. 02000000:00000001:5.0:1496184950.433004:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433005:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.433008:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.433009:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.433011:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.433012:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.433014:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.433016:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.433018:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.433019:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.433020:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.433021:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.433022:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.433023:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.433024:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.433026:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.433027:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.433028:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.433029:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.433030:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.433031:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.433035:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.433035:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.433037:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.433039:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.433039:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.433041:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.433043:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.433043:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.433045:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.433046:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.433048:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.433049:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433050:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.433051:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.433053:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.433054:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433055:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.433056:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.433058:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.433059:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.433060:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.433061:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.433062:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.433064:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.433065:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433066:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.433069:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.433071:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.433072:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.433074:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.433075:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.433076:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433077:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.433079:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.433080:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.433081:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.433082:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.433084:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433085:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.433086:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.433087:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.433088:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.433089:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.433090:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.433091:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.433093:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.433094:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.433095:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.433096:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433098:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.433099:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.433100:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.433101:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.433102:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.433103:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.433104:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433105:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.433106:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.433107:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107408 00010000:00000001:5.0:1496184950.433109:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.433111:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdec50 x1568861104107408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.433116:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.433117:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.433118:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.433120:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.433122:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.433123:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.433125:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.433126:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.433127:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433128:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.433130:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.433132:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107408, offset 0 00000100:00000001:5.0:1496184950.433142:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433144:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.433145:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.433146:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.433147:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.433148:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433149:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.433151:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdec50 x1568861104107408/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.433156:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107408:12345-10.37.248.124@o2ib1:8 Request procesed in 168us (274us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.433160:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43429 00000100:00000040:5.0:1496184950.433162:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.433163:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.433164:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.433165:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7c00. 00000020:00000010:5.0:1496184950.433167:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.433169:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.433170:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.433171:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.433173:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.433177:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.433177:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.433241:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.433243:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.433244:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68ac00. 02000000:00000001:7.0:1496184950.433247:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.433248:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.433669:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.433671:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.433673:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.433678:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.433685:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.433687:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.433720:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.433723:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.433724:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.433725:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.433727:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.433728:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.433730:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.433732:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.433734:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.433734:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.433735:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.433737:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.433738:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.433740:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.433742:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.433743:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.433745:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.433747:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7b80. 00000020:00000010:5.0:1496184950.433748:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.433751:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.433752:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.433753:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.433754:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.433755:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.433756:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.433758:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.433759:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.433760:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.433761:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.433762:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.433763:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.433764:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.433765:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.433766:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.433766:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433767:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.433769:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433770:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433771:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433772:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.433773:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433774:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.433777:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.433784:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.433784:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.433786:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.433787:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.433788:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.433790:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.433791:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.433792:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.433792:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.433793:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00002000:00000001:5.0:1496184950.433794:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433795:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433798:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433802:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.433804:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.433805:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.433806:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.433807:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.433807:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.433808:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.433811:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57702 00000100:00000040:5.0:1496184950.433812:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.433814:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671270480 : -131872038281136 : ffff88102600a050) 00000100:00000040:5.0:1496184950.433816:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.433821:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.433822:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.433824:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.433826:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.433827:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.433828:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.433829:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433830:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.433831:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.433832:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.433833:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.433834:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.433835:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.433836:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433837:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.433839:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.433840:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.433841:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.433842:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433844:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433845:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.433846:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.433847:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.433852:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.433854:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.433855:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.433857:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.433859:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.433860:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.433861:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.433863:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.433864:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.433865:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.433867:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.433869:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.433869:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.433870:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.433871:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.433873:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.433873:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.433874:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.433877:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.433910:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.433917:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433918:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.433921:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433922:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.433923:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.433924:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.433925:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.433927:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.433928:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.433930:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.433934:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.433935:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.433961:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfefa0-0x592def0cfefa0 00000100:00000001:5.0:1496184950.433963:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.435371:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.435385:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.446416:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.446423:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.446424:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.457457:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:2.0:1496184950.457488:0:109:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:2.0:1496184950.457491:0:109:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:2.0:1496184950.457494:0:109:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89c50 x1568861104107440 msgsize 520 00000100:00100000:2.0:1496184950.457501:0:109:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:2.0:1496184950.457502:0:109:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:2.0:1496184950.457506:0:109:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.457512:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.457515:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.457519:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 00010000:00020000:5.0:1496184950.457521:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 02000000:00000001:3.0:1496184950.457521:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.457532:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184950.457534:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184950.457538:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.457540:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.457542:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00100000:3.0:1496184950.457542:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107440 00000020:00000001:3.0:1496184950.457545:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:5.0:1496184950.457546:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00000020:00000040:3.0:1496184950.457546:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00000001:5.0:1496184950.457547:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000020:00000001:3.0:1496184950.457548:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00002000:00000001:5.0:1496184950.457549:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.457550:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000020:00000040:3.0:1496184950.457551:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:3.0:1496184950.457553:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:3.0:1496184950.457555:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:3.0:1496184950.457559:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.457560:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:3.0:1496184950.457563:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000010:3.0:1496184950.457566:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8900. 00000020:00000010:3.0:1496184950.457569:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2000. 00000100:00000001:3.0:1496184950.457572:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:3.0:1496184950.457573:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:3.0:1496184950.457575:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184950.457580:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000001:00000001:5.0:1496184950.457584:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000200:3.0:1496184950.457585:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b89c50 x1568861104107440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000002:5.0:1496184950.457586:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.457588:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.457589:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.457590:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184950.457590:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00002000:00000001:5.0:1496184950.457591:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.457592:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:3.0:1496184950.457592:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.457593:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.457594:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.457595:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000100:00000001:3.0:1496184950.457595:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00000020:00000040:5.0:1496184950.457596:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000010:5.0:1496184950.457598:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.457599:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00004000:00000001:3.0:1496184950.457599:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.457600:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:5.0:1496184950.457601:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 00004000:00000001:3.0:1496184950.457601:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 02000000:00000001:5.0:1496184950.457602:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 00004000:00000001:3.0:1496184950.457602:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 02000000:00000010:5.0:1496184950.457603:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68a400. 00000100:00000001:3.0:1496184950.457604:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 02000000:00000001:5.0:1496184950.457605:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.457607:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000100:00000001:3.0:1496184950.457607:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000020:00000001:5.0:1496184950.457608:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00100000:3.0:1496184950.457610:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6112 00010000:00000080:5.0:1496184950.457611:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000040:3.0:1496184950.457612:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 2 00000100:00000001:3.0:1496184950.457613:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736387152 : -131906973164464 : ffff880803b89c50) 00010000:00000040:5.0:1496184950.457616:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000040:3.0:1496184950.457616:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89c50 x1568861104107440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00010000:00000001:5.0:1496184950.457618:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.457619:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.457619:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.457621:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000001:3.0:1496184950.457622:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000040:5.0:1496184950.457623:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.457623:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184950.457625:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+9:12082:x1568861104107440:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184950.457628:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107440 00000100:00100000:5.0:1496184950.457629:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23806us (23954us total) trans 0 rc -110/-999 00000020:00000001:3.0:1496184950.457630:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000100:00100000:5.0:1496184950.457634:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57702 00010000:00000001:3.0:1496184950.457634:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000100:00000040:5.0:1496184950.457636:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.457637:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.457638:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.457641:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000020:00000040:3.0:1496184950.457641:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00002000:00000001:5.0:1496184950.457646:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.457647:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a050 with x1568861102817424 ext(0->1048575) 00000100:00000001:3.0:1496184950.457647:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184950.457648:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 00010000:00000001:5.0:1496184950.457649:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.457650:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.457651:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 02000000:00000010:3.0:1496184950.457651:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23f8400. 02000000:00000001:3.0:1496184950.457652:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.457653:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.457654:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00000100:00000001:3.0:1496184950.457654:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.457655:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.457656:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.457657:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a050. 00002000:00000001:5.0:1496184950.457658:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.457659:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000040:3.0:1496184950.457660:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000010:5.0:1496184950.457661:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7b80. 00010000:00000001:3.0:1496184950.457662:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00000020:00000010:5.0:1496184950.457663:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.457664:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00010000:02000400:3.0:1496184950.457665:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00000020:00000040:5.0:1496184950.457666:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184950.457667:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00000100:00000001:5.0:1496184950.457668:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00010000:00000001:3.0:1496184950.457669:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00000100:00000200:5.0:1496184950.457670:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00010000:00080000:3.0:1496184950.457672:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00004000:00000001:5.0:1496184950.457675:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.457676:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00000001:3.0:1496184950.457676:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.457678:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.457681:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.457683:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.457685:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.457686:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.457688:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.457690:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.457693:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.457694:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.457696:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.457697:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.457699:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.457704:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.457705:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.457707:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.457709:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.457710:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.457720:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.457722:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.457723:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.457724:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.457727:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.457728:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.457730:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.457731:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.457733:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.457735:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.457737:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.457738:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.457739:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.457741:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.457743:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.457744:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.457745:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.457748:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.457750:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.457752:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.457754:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.457757:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.457759:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.457762:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.457765:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.457766:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.457767:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.457768:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.457770:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.457771:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.457773:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.457775:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.457777:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.457779:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.457781:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.457782:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.457784:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.457786:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.457787:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.457788:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.457791:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.457792:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.457794:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.457795:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.457798:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.457801:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.457802:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.457803:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.457805:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.457806:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.457807:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.457809:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.457811:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.457814:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107440 00010000:00000001:3.0:1496184950.457816:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.457819:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b89c50 x1568861104107440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.457826:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.457827:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.457829:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.457832:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.457834:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.457835:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.457837:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.457838:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.457839:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.457841:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.457843:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.457847:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107440, offset 0 00000100:00000001:3.0:1496184950.457872:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.457873:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.457874:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.457875:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.457876:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.457877:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.457878:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.457881:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89c50 x1568861104107440/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.457886:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107440:12345-10.37.248.124@o2ib1:8 Request procesed in 263us (391us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.457901:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6112 00000100:00000040:3.0:1496184950.457903:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.457905:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.457906:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.457909:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8900. 00000020:00000010:3.0:1496184950.457912:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2000. 00000020:00000010:3.0:1496184950.457915:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88085de99200. 00000020:00000040:3.0:1496184950.457916:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.457919:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.457921:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.457927:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.457928:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.458272:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.458276:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.458289:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23f8400. 02000000:00000001:6.0:1496184950.458292:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.458293:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.458684:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.458685:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.458688:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026070450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.458693:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.458694:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.458696:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.458730:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.458732:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.458734:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.458735:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.458736:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.458737:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.458740:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.458742:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.458744:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.458745:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.458746:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.458747:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.458749:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.458750:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.458753:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.458754:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.458756:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.458759:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8800742d7100. 00000020:00000010:5.0:1496184950.458760:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.458762:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.458764:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.458765:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.458766:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.458767:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.458769:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.458770:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.458771:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.458773:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.458774:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.458775:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.458776:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.458777:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.458778:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.458778:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.458779:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458780:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.458782:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458783:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458784:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.458785:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.458786:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.458787:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.458790:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.458796:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.458797:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.458799:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.458800:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.458801:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.458802:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.458804:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.458804:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.458805:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.458806:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00002000:00000001:5.0:1496184950.458807:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.458808:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.458810:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.458815:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.458817:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.458818:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.458819:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.458820:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.458821:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.458822:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.458825:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57703 00000100:00000040:5.0:1496184950.458827:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.458828:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671689296 : -131872037862320 : ffff881026070450) 00000100:00000040:5.0:1496184950.458830:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.458835:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.458836:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.458838:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.458840:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.458841:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.458843:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.458844:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458845:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.458845:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.458847:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.458848:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.458849:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.458850:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.458851:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458852:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.458854:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.458855:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.458856:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.458857:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.458859:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458860:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.458861:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.458863:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.458871:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.458873:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.458874:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.458876:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.458878:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.458879:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.458880:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.458882:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.458884:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.458885:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.458887:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.458888:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.458889:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.458902:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.458903:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.458905:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.458905:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.458906:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.458909:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.458929:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.458935:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458936:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.458939:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.458940:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.458941:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.458942:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.458943:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.458945:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.458946:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.458947:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.458952:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.458953:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.458980:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfefc0-0x592def0cfefc0 00000100:00000001:5.0:1496184950.458982:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.470149:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.470151:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.481133:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.481138:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.481139:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.492116:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.492178:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.492197:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.492199:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.492203:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.492204:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.492206:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.492208:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.492241:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.492243:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.492245:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.492246:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.492247:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.492248:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.492249:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.492251:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.492251:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.492252:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.492253:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.492255:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.492257:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.492258:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.492259:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.492260:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.492261:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.492263:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.492265:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.492267:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.492272:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.492274:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.492275:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.492276:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.492277:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.492279:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.492285:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 33448us (33595us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.492290:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57703 00000100:00000040:5.0:1496184950.492292:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.492293:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.492294:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.492297:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026070450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.492302:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.492303:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026070450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.492305:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.492305:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.492307:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.492308:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.492310:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.492310:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.492311:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.492311:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026070450. 00002000:00000001:5.0:1496184950.492313:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.492314:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.492315:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8800742d7100. 00000020:00000010:5.0:1496184950.492317:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.492319:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.492320:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.492322:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.492324:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.492329:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.492331:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.503062:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.503064:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.503067:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf050 x1568861104107472 msgsize 520 00000100:00100000:6.0:1496184950.503072:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.503073:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.503075:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.503107:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.503110:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.503112:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.503113:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.503114:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.503115:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.503118:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.503119:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107472 00000020:00000001:5.0:1496184950.503121:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.503122:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.503123:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.503124:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.503126:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.503127:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.503129:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.503130:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.503132:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.503135:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085de54180. 00000020:00000010:5.0:1496184950.503136:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.503138:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.503138:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.503139:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.503141:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.503145:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf050 x1568861104107472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.503150:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.503151:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503155:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.503158:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.503158:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.503159:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.503160:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.503161:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.503162:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.503164:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43430 00000100:00000040:5.0:1496184950.503166:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.503167:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683677264 : -131872025874352 : ffff881026bdf050) 00000100:00000040:5.0:1496184950.503169:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf050 x1568861104107472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.503174:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.503175:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.503177:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107472:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.503179:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107472 00000020:00000001:5.0:1496184950.503180:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.503182:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.503185:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.503188:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.503189:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.503190:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68a400. 02000000:00000001:5.0:1496184950.503191:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503193:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.503195:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.503197:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.503199:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.503200:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.503202:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.503204:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.503207:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.503208:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.503209:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.503210:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.503212:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.503212:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.503214:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.503215:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.503217:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.503218:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.503219:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.503220:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.503221:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.503224:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.503225:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.503226:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.503228:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.503229:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.503231:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.503233:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.503233:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.503234:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.503236:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.503238:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.503240:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.503241:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.503242:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.503243:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.503245:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.503246:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.503247:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.503249:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.503250:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.503251:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.503252:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.503253:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.503255:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.503256:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503257:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.503260:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.503263:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.503265:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.503267:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.503268:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.503269:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503270:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.503272:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.503273:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.503274:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.503276:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.503277:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503278:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.503279:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.503280:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.503281:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.503283:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.503284:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.503285:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.503287:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.503288:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.503289:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.503290:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503292:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.503293:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.503295:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.503295:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.503296:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.503297:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.503298:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503299:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.503301:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.503302:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107472 00010000:00000001:5.0:1496184950.503303:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.503305:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf050 x1568861104107472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.503310:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.503311:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.503313:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.503315:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.503317:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.503318:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.503319:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.503320:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.503321:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503322:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.503324:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.503326:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107472, offset 0 00000100:00000001:5.0:1496184950.503341:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503342:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.503343:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.503344:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.503346:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.503346:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503347:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.503349:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf050 x1568861104107472/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.503354:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107472:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (286us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.503359:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43430 00000100:00000040:5.0:1496184950.503360:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.503361:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.503362:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.503364:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085de54180. 00000020:00000010:5.0:1496184950.503365:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.503367:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.503368:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.503370:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.503372:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.503375:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.503376:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.503431:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.503433:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.503435:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68a400. 02000000:00000001:6.0:1496184950.503438:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.503439:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.503804:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.503806:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.503808:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026009c50 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.503813:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.503814:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.503816:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.503847:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.503850:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.503851:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.503852:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.503853:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.503854:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.503857:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.503858:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.503860:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.503861:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.503862:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.503863:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.503865:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.503866:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.503868:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.503869:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.503871:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.503873:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085de54c00. 00000020:00000010:5.0:1496184950.503874:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.503876:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.503878:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.503878:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.503879:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.503880:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.503881:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.503883:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.503884:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.503885:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.503886:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.503887:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.503888:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.503901:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.503902:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.503903:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.503904:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503905:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.503906:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503907:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503908:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.503909:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.503910:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.503912:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.503915:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.503921:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.503922:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.503924:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.503924:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.503925:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.503927:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.503928:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.503929:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.503929:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.503930:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00002000:00000001:5.0:1496184950.503931:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503932:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503935:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503939:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.503941:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.503942:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.503943:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.503943:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.503944:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.503945:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.503948:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57704 00000100:00000040:5.0:1496184950.503949:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.503950:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671269456 : -131872038282160 : ffff881026009c50) 00000100:00000040:5.0:1496184950.503953:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.503958:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.503959:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.503960:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.503963:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.503964:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.503965:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.503966:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503967:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.503968:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.503969:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.503971:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.503972:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.503972:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.503973:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503974:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.503976:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.503977:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.503978:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.503979:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.503980:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.503982:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.503982:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.503984:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.503989:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.503991:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.503992:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.503994:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.503996:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.503997:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.503998:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.503999:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.504001:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.504002:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.504004:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.504005:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.504006:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.504007:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.504007:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.504009:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.504010:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.504011:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.504013:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.504035:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.504040:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.504042:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.504044:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.504046:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.504047:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.504048:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.504049:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.504050:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.504051:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.504053:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.504058:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.504059:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.504090:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cfefe0-0x592def0cfefe0 00000100:00000001:5.0:1496184950.504092:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.505689:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.505692:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.505700:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.505706:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.505707:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.505713:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.505760:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.505767:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.505769:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.505773:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.505774:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.505776:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.505778:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.505811:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.505813:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.505815:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.505816:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.505817:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.505818:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.505819:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.505820:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.505821:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.505822:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.505823:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.505824:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.505826:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.505827:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.505828:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.505829:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.505831:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.505832:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.505834:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.505837:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.505842:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.505843:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.505844:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.505845:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.505846:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.505848:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.505854:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1894us (2044us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.505858:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57704 00000100:00000040:5.0:1496184950.505860:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.505862:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.505862:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.505865:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009c50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.505871:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.505871:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009c50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.505873:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.505874:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.505875:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.505876:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.505878:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.505879:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.505879:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.505880:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009c50. 00002000:00000001:5.0:1496184950.505881:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.505882:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.505884:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085de54c00. 00000020:00000010:5.0:1496184950.505885:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.505887:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.505888:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.505898:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.505901:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.505906:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.505907:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.534045:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.534047:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.534049:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf450 x1568861104107504 msgsize 520 00000100:00100000:6.0:1496184950.534055:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.534056:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.534058:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.534088:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.534090:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.534092:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.534093:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.534094:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.534095:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.534097:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.534099:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107504 00000020:00000001:5.0:1496184950.534101:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.534102:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.534103:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.534104:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.534105:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.534107:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.534109:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.534110:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.534111:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.534114:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085de54280. 00000020:00000010:5.0:1496184950.534115:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.534117:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.534117:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.534118:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.534120:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.534124:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf450 x1568861104107504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.534129:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.534129:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534134:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.534136:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.534137:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.534137:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.534138:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.534139:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.534140:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.534142:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43431 00000100:00000040:5.0:1496184950.534143:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.534145:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683678288 : -131872025873328 : ffff881026bdf450) 00000100:00000040:5.0:1496184950.534147:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf450 x1568861104107504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.534152:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.534153:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.534155:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107504:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.534158:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107504 00000020:00000001:5.0:1496184950.534159:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.534161:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.534164:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.534167:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.534168:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.534169:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68b400. 02000000:00000001:5.0:1496184950.534170:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534171:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.534174:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.534175:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.534177:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.534179:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.534180:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.534182:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.534185:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.534186:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.534187:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.534188:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.534189:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.534190:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.534191:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.534192:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.534194:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.534194:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.534195:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.534196:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.534198:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.534201:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.534202:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.534203:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.534206:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.534207:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.534209:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.534210:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.534211:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.534212:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.534214:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.534215:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.534217:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.534218:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.534218:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.534220:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.534222:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.534222:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.534223:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.534225:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.534226:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.534227:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.534228:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.534229:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.534231:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.534232:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534233:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.534236:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.534238:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.534239:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.534241:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.534242:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.534243:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534244:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.534246:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.534247:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.534248:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.534249:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.534251:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534252:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.534253:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.534254:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.534255:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.534256:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.534257:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.534258:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.534260:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.534261:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.534262:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.534263:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534265:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.534266:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.534267:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.534268:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.534269:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.534270:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.534271:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534272:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.534273:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.534274:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107504 00010000:00000001:5.0:1496184950.534276:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.534278:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf450 x1568861104107504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.534283:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.534284:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.534285:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.534287:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.534289:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.534290:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.534292:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.534293:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.534294:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534295:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.534297:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.534299:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107504, offset 0 00000100:00000001:5.0:1496184950.534310:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534311:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.534312:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.534313:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.534314:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.534315:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534316:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.534318:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf450 x1568861104107504/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.534323:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107504:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (272us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.534327:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43431 00000100:00000040:5.0:1496184950.534329:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.534330:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.534331:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.534333:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085de54280. 00000020:00000010:5.0:1496184950.534335:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.534336:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.534337:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.534339:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.534341:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.534344:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.534345:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.534395:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.534398:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.534399:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68b400. 02000000:00000001:7.0:1496184950.534402:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.534403:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.534853:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.534855:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.534857:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.534862:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.534863:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.534865:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.534906:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.534908:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.534910:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.534911:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.534913:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.534914:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.534916:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.534918:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.534919:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.534920:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.534921:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.534922:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.534924:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.534925:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.534928:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.534928:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.534930:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.534932:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8b80. 00000020:00000010:5.0:1496184950.534933:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.534935:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.534937:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.534937:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.534938:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.534940:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.534940:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.534942:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.534943:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.534944:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.534945:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.534946:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.534947:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.534948:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.534949:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.534950:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.534950:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534952:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.534953:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534954:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.534955:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.534956:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.534957:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.534966:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.534969:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.534975:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.534976:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.534978:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.534979:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.534980:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.534981:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.534983:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.534983:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.534984:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.534984:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00002000:00000001:5.0:1496184950.534986:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534987:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534989:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.534993:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.534995:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.534996:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.534997:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.534998:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.534999:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.535000:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.535002:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57705 00000100:00000040:5.0:1496184950.535004:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.535005:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671274576 : -131872038277040 : ffff88102600b050) 00000100:00000040:5.0:1496184950.535007:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.535012:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.535013:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.535015:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.535017:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.535018:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.535020:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.535021:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.535022:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.535027:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.535029:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.535030:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.535031:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.535031:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.535032:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.535033:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.535035:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.535036:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.535037:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.535038:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.535040:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.535041:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.535042:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.535044:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.535048:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.535050:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.535051:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.535053:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.535055:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.535056:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.535057:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.535058:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.535060:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.535061:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.535063:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.535065:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.535065:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.535066:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.535067:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.535069:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.535069:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.535070:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.535073:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.535093:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.535098:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.535100:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.535102:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.535104:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.535105:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.535106:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.535107:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.535108:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.535110:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.535111:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.535116:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.535117:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.535143:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff000-0x592def0cff000 00000100:00000001:5.0:1496184950.535145:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.536796:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.536810:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.536816:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.536821:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.536821:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.536827:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.536865:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.536871:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.536877:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.536881:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.536882:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.536884:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.536885:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.536927:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.536929:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.536931:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.536932:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.536932:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.536934:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.536935:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.536937:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.536937:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.536938:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.536939:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.536941:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.536943:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.536944:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.536945:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.536946:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.536947:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.536949:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.536951:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.536953:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.536958:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.536960:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.536961:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.536961:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.536963:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.536964:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.536970:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1956us (2112us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.536975:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57705 00000100:00000040:5.0:1496184950.536977:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.536979:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.536980:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.536982:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.536988:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.536988:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.536990:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.536991:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.536992:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.536994:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.536995:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.536996:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.536996:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.536997:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b050. 00002000:00000001:5.0:1496184950.536998:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.536999:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.537001:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8b80. 00000020:00000010:5.0:1496184950.537002:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.537004:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.537005:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.537007:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.537009:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.537014:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.537015:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.565617:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.565619:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.565621:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff881026bdf850 x1568861104107536 msgsize 520 00000100:00100000:6.0:1496184950.565626:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.565627:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.565630:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.565640:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.565642:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.565644:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.565645:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.565646:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.565647:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.565650:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.565652:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107536 00000020:00000001:5.0:1496184950.565653:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.565654:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.565655:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.565656:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.565658:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.565659:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.565661:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.565662:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.565664:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.565666:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8d00. 00000020:00000010:5.0:1496184950.565675:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.565677:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.565678:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.565679:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.565681:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.565684:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff881026bdf850 x1568861104107536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.565689:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.565690:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565694:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.565696:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.565697:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.565698:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.565699:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.565700:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.565700:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.565703:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43432 00000100:00000040:5.0:1496184950.565704:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.565705:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201683679312 : -131872025872304 : ffff881026bdf850) 00000100:00000040:5.0:1496184950.565708:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026bdf850 x1568861104107536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.565713:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.565714:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.565715:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107536:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.565717:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107536 00000020:00000001:5.0:1496184950.565718:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.565721:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.565724:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.565726:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.565727:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.565728:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68d000. 02000000:00000001:5.0:1496184950.565730:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565731:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.565742:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.565744:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.565746:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.565747:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.565749:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.565751:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.565753:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.565754:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.565755:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.565756:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.565757:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.565758:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.565759:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.565760:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.565762:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.565763:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.565764:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.565765:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.565766:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.565769:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.565770:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.565771:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.565773:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.565774:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.565776:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.565777:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.565778:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.565779:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.565781:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.565782:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.565784:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.565785:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.565786:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.565788:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.565789:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.565790:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.565791:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.565792:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.565794:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.565795:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.565796:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.565797:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.565798:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.565800:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565801:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.565803:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.565805:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.565807:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.565808:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.565810:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.565811:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565812:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.565813:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.565814:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.565816:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.565817:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.565818:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565819:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.565820:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.565821:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.565822:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.565824:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.565824:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.565826:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.565827:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.565829:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.565830:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.565831:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.565832:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.565833:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.565835:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.565836:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.565837:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.565837:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.565838:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.565839:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.565841:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.565842:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107536 00010000:00000001:5.0:1496184950.565843:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.565845:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff881026bdf850 x1568861104107536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.565850:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.565851:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.565852:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.565854:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.565856:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.565857:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.565859:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.565860:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.565861:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565862:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.565863:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.565865:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107536, offset 0 00000100:00000001:5.0:1496184950.565876:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.565877:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.565879:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.565880:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.565881:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.565882:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.565883:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.565884:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026bdf850 x1568861104107536/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.565901:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107536:12345-10.37.248.124@o2ib1:8 Request procesed in 176us (267us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.565906:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43432 00000100:00000040:5.0:1496184950.565907:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.565908:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.565909:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.565911:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8d00. 00000020:00000010:5.0:1496184950.565912:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.565914:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.565916:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.565917:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.565919:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.565922:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.565923:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.565939:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.565941:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.565943:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68d000. 02000000:00000001:6.0:1496184950.565946:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.565947:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.566386:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.566388:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.566390:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600a450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.566395:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.566396:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.566398:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.566428:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.566430:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.566432:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.566433:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.566434:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.566435:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.566438:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.566439:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.566441:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.566442:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.566443:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.566444:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.566446:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.566447:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.566449:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.566450:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.566452:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.566454:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8f00. 00000020:00000010:5.0:1496184950.566455:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.566458:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.566459:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.566460:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.566461:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.566462:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.566463:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.566465:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.566466:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.566467:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.566468:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.566469:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.566470:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.566471:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.566472:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.566473:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.566474:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566475:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.566476:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566477:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566478:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.566479:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.566480:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.566481:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.566484:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.566491:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.566492:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.566493:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.566494:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.566495:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.566497:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.566498:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.566499:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.566499:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.566500:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00002000:00000001:5.0:1496184950.566501:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.566502:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.566504:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.566509:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.566511:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.566512:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.566513:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.566514:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.566515:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.566515:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.566518:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57706 00000100:00000040:5.0:1496184950.566519:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.566521:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671271504 : -131872038280112 : ffff88102600a450) 00000100:00000040:5.0:1496184950.566523:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.566528:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.566529:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.566530:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.566532:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.566533:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.566535:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.566536:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566537:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.566538:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.566539:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.566540:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.566541:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.566542:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.566543:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566544:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.566546:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.566546:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.566548:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.566549:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.566550:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566552:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.566552:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.566554:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.566558:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.566561:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.566562:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.566564:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.566566:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.566567:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.566568:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.566569:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.566571:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.566572:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.566574:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.566575:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.566576:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.566576:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.566577:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.566579:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.566580:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.566581:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.566583:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.566605:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.566610:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566612:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.566614:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.566616:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.566617:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.566618:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.566619:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.566620:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.566621:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.566623:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.566627:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.566628:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.566654:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff020-0x592def0cff020 00000100:00000001:5.0:1496184950.566657:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.568455:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.568458:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.568475:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.568480:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.568480:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.568485:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.568523:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.568530:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.568532:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.568536:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.568537:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.568539:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.568541:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.568574:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.568576:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.568578:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.568579:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.568579:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.568581:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.568582:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.568583:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.568584:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.568585:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.568586:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.568587:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.568589:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.568590:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.568591:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.568593:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.568594:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.568596:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.568597:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.568600:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.568605:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.568607:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.568607:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.568608:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.568609:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.568611:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.568617:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2088us (2225us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.568622:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57706 00000100:00000040:5.0:1496184950.568624:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.568625:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.568626:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.568629:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600a450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.568634:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.568634:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600a450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.568636:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.568637:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.568638:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.568640:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.568641:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.568642:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.568642:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.568643:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600a450. 00002000:00000001:5.0:1496184950.568644:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.568646:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.568647:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8f00. 00000020:00000010:5.0:1496184950.568649:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.568650:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.568652:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.568653:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.568655:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.568660:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.568661:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.596308:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.596310:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.596312:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c9050 x1568861104107568 msgsize 520 00000100:00100000:7.0:1496184950.596317:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.596318:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.596320:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.596353:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.596356:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.596358:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.596359:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.596360:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.596361:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.596363:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.596365:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107568 00000020:00000001:5.0:1496184950.596366:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.596367:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.596368:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.596369:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.596371:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.596373:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.596375:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.596375:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.596377:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.596379:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8600. 00000020:00000010:5.0:1496184950.596381:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.596382:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.596383:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.596384:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.596386:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.596389:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c9050 x1568861104107568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.596398:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.596399:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596403:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.596405:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.596406:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.596407:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.596408:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.596409:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.596409:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.596412:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43433 00000100:00000040:5.0:1496184950.596413:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.596414:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677295696 : -131872032255920 : ffff8810265c9050) 00000100:00000040:5.0:1496184950.596417:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c9050 x1568861104107568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.596422:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.596423:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.596424:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107568:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.596427:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107568 00000020:00000001:5.0:1496184950.596428:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.596430:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.596433:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.596435:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.596436:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.596438:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68a800. 02000000:00000001:5.0:1496184950.596439:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596440:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.596442:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.596444:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.596446:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.596447:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.596449:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.596451:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.596453:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.596454:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.596455:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.596456:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.596457:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.596458:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.596459:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.596460:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.596462:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.596463:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.596464:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.596465:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.596466:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.596469:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.596470:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.596471:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.596473:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.596474:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.596476:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.596478:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.596478:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.596479:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.596481:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.596483:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.596484:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.596485:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.596486:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.596487:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.596489:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.596490:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.596491:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.596492:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.596494:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.596495:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.596495:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.596497:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.596498:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.596500:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596501:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.596503:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.596505:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.596507:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.596508:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.596509:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.596510:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596511:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.596513:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.596514:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.596515:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.596517:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.596518:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596519:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.596520:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.596521:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.596522:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.596523:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.596524:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.596525:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.596527:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.596528:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.596529:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.596530:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.596532:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.596533:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.596534:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.596535:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.596536:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.596537:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.596538:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.596539:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.596540:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.596541:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107568 00010000:00000001:5.0:1496184950.596543:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.596545:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c9050 x1568861104107568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.596550:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.596551:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.596552:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.596554:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.596556:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.596557:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.596558:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.596559:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.596560:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596561:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.596563:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.596565:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107568, offset 0 00000100:00000001:5.0:1496184950.596575:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.596576:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.596578:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.596579:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.596580:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.596581:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.596582:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.596583:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c9050 x1568861104107568/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.596589:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107568:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (275us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.596593:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43433 00000100:00000040:5.0:1496184950.596595:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.596596:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.596597:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.596598:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8600. 00000020:00000010:5.0:1496184950.596600:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.596601:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.596603:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.596604:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.596606:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.596609:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.596610:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.596671:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.596673:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.596675:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68a800. 02000000:00000001:7.0:1496184950.596678:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.596678:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.597185:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.597187:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.597197:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff881026009850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.597202:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.597203:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.597205:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.597235:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.597237:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.597239:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.597240:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.597241:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.597242:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.597244:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.597246:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.597248:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.597248:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.597250:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.597251:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.597252:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.597254:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.597256:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.597257:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.597258:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.597260:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8200. 00000020:00000010:5.0:1496184950.597262:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.597264:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.597265:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.597266:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.597267:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.597268:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.597269:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.597271:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.597272:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.597273:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.597274:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.597275:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.597276:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.597277:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.597278:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.597279:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.597279:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597281:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.597282:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597283:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597284:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.597285:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.597286:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.597287:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.597290:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.597297:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.597297:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.597299:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.597300:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.597301:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.597302:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.597304:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.597304:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.597305:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.597305:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00002000:00000001:5.0:1496184950.597307:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.597308:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.597310:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.597314:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.597316:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.597317:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.597318:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.597319:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.597320:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.597321:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.597323:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57707 00000100:00000040:5.0:1496184950.597325:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.597326:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671268432 : -131872038283184 : ffff881026009850) 00000100:00000040:5.0:1496184950.597328:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.597333:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.597334:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.597336:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.597338:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.597339:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.597340:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.597341:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597342:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.597343:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.597344:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.597345:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.597346:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.597347:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.597348:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597349:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.597351:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.597351:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.597353:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.597354:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.597355:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597356:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.597357:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.597359:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.597363:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.597366:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.597367:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.597369:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.597371:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.597372:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.597373:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.597374:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.597376:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.597376:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.597379:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.597380:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.597381:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.597381:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.597382:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.597384:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.597385:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.597386:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.597388:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.597410:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.597416:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597417:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.597420:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.597421:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.597422:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.597423:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.597424:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.597426:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.597427:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.597429:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.597433:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.597434:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.597461:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff040-0x592def0cff040 00000100:00000001:5.0:1496184950.597463:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.598810:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.598811:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.609919:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.609922:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.609923:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:2.0:1496184950.613927:0:13688:0:(client.c:2174:ptlrpc_expired_set()) Process entered 00000100:00000001:2.0:1496184950.613929:0:13688:0:(client.c:2209:ptlrpc_expired_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1496184950.613936:0:13688:0:(client.c:2262:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1496184950.613938:0:13688:0:(client.c:2298:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184950.613940:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184950.613943:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1496184950.613945:0:13688:0:(ptlrpcd.c:321:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1496184950.613946:0:13688:0:(ptlrpcd.c:416:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.620901:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.620982:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.620989:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.620991:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.620995:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.620996:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.620998:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.621000:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.621033:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.621035:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.621037:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.621038:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.621039:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.621040:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.621041:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.621042:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.621043:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.621044:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.621045:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.621047:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.621048:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.621050:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.621050:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.621051:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.621053:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.621054:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.621056:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.621059:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.621064:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.621065:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.621066:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.621067:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.621068:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.621070:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.621076:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23741us (23877us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.621081:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57707 00000100:00000040:5.0:1496184950.621083:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.621084:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.621085:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.621088:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff881026009850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.621093:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.621094:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff881026009850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.621096:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.621096:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.621098:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.621099:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.621100:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.621101:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.621102:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.621102:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff881026009850. 00002000:00000001:5.0:1496184950.621103:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.621105:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.621106:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8200. 00000020:00000010:5.0:1496184950.621108:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.621109:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.621111:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.621112:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.621114:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.621120:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.621121:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.631639:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.631641:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.631644:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8c50 x1568861104107600 msgsize 520 00000100:00100000:7.0:1496184950.631649:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.631650:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.631652:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.631684:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.631687:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.631688:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.631689:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.631690:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.631691:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.631694:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.631696:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107600 00000020:00000001:5.0:1496184950.631697:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.631698:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.631699:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.631701:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.631702:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.631704:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.631706:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.631707:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.631708:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.631710:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8180. 00000020:00000010:5.0:1496184950.631711:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.631713:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.631714:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.631715:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.631717:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.631720:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8c50 x1568861104107600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.631725:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.631726:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631731:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.631733:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.631733:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.631734:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.631735:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.631736:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.631736:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.631739:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43434 00000100:00000040:5.0:1496184950.631741:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.631742:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677294672 : -131872032256944 : ffff8810265c8c50) 00000100:00000040:5.0:1496184950.631744:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8c50 x1568861104107600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.631749:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.631750:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.631752:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107600:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.631754:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107600 00000020:00000001:5.0:1496184950.631755:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.631757:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.631760:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.631762:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.631763:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.631765:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68cc00. 02000000:00000001:5.0:1496184950.631766:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631767:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.631769:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.631771:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.631773:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.631774:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.631776:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.631778:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.631780:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.631781:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.631782:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.631783:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.631784:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.631785:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.631786:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.631787:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.631789:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.631790:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.631791:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.631792:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.631793:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.631797:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.631797:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.631799:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.631801:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.631801:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.631803:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.631805:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.631806:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.631807:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.631808:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.631810:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.631811:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.631812:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.631813:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.631815:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.631816:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.631817:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.631818:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.631820:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.631821:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.631822:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.631823:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.631824:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.631826:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.631827:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631828:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.631830:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.631833:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.631835:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.631836:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.631838:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.631839:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631840:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.631841:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.631842:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.631844:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.631845:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.631846:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631847:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.631848:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.631849:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.631850:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.631852:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.631852:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.631854:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.631855:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.631856:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.631858:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.631859:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.631860:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.631861:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.631863:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.631864:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.631865:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.631866:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.631866:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.631867:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.631869:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.631870:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107600 00010000:00000001:5.0:1496184950.631871:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.631873:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8c50 x1568861104107600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.631879:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.631879:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.631881:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.631883:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.631884:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.631886:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.631887:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.631888:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.631889:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631903:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.631905:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.631907:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107600, offset 0 00000100:00000001:5.0:1496184950.631918:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.631919:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.631920:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.631921:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.631922:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.631923:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.631924:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.631926:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8c50 x1568861104107600/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.631932:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107600:12345-10.37.248.124@o2ib1:8 Request procesed in 181us (286us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.631936:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43434 00000100:00000040:5.0:1496184950.631938:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.631939:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.631940:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.631941:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8180. 00000020:00000010:5.0:1496184950.631943:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.631944:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.631946:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.631947:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.631949:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.631953:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.631953:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.632018:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.632022:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.632024:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68cc00. 02000000:00000001:7.0:1496184950.632028:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.632029:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.632460:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.632462:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.632465:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600ac50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.632470:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.632476:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.632478:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.632512:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.632515:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.632517:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.632518:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.632519:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.632520:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.632522:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.632524:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.632526:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.632527:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.632528:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.632529:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.632530:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.632532:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.632534:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.632535:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.632537:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.632539:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8d80. 00000020:00000010:5.0:1496184950.632540:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.632542:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.632544:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.632544:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.632545:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.632546:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.632547:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.632549:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.632550:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.632551:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.632552:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.632553:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.632554:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.632555:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.632556:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.632557:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.632558:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632559:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.632560:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632561:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632562:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.632563:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.632564:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.632565:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.632568:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.632575:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.632575:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.632577:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.632578:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.632579:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.632580:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.632582:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.632582:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.632583:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.632584:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00002000:00000001:5.0:1496184950.632585:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.632586:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.632588:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.632593:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.632595:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.632596:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.632597:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.632597:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.632598:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.632599:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.632602:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57708 00000100:00000040:5.0:1496184950.632603:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.632604:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671273552 : -131872038278064 : ffff88102600ac50) 00000100:00000040:5.0:1496184950.632607:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.632612:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.632613:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.632614:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.632616:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.632617:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.632619:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.632620:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632621:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.632622:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.632623:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.632624:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.632625:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.632626:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.632627:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632628:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.632630:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.632630:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.632632:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.632633:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.632634:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632636:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.632636:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.632638:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.632642:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.632645:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.632646:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.632648:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.632650:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.632651:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.632652:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.632653:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.632655:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.632656:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.632658:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.632659:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.632660:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.632660:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.632661:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.632663:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.632664:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.632665:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.632667:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.632691:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.632697:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632698:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.632701:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.632702:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.632703:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.632704:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.632705:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.632706:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.632708:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.632709:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.632714:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.632715:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.632741:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff060-0x592def0cff060 00000100:00000001:5.0:1496184950.632743:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.634315:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.634317:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.645347:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.645352:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.645353:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.656380:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.656446:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.656465:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000100:00000001:1.0:1496184950.656466:0:21159:0:(events.c:294:request_in_callback()) Process entered 00000020:00000001:5.0:1496184950.656467:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000200:1.0:1496184950.656469:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:5.0:1496184950.656471:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.656472:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000040:1.0:1496184950.656472:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b89850 x1568861104107632 msgsize 520 00002000:00000001:5.0:1496184950.656474:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.656476:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184950.656479:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184950.656480:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184950.656483:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.656489:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.656492:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.656496:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184950.656498:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.656500:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184950.656502:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184950.656506:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.656509:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184950.656510:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107632 00000020:00000002:5.0:1496184950.656511:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.656513:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00000020:00000001:3.0:1496184950.656513:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:5.0:1496184950.656514:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00000020:00000040:3.0:1496184950.656514:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00002000:00000001:5.0:1496184950.656515:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.656516:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.656517:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1496184950.656517:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.656519:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000040:3.0:1496184950.656519:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000020:00000001:5.0:1496184950.656520:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.656520:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000001:3.0:1496184950.656521:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000040:5.0:1496184950.656522:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 9 00000100:00000010:5.0:1496184950.656523:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000001:3.0:1496184950.656523:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000010:5.0:1496184950.656525:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.656526:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.656527:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.656528:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68e400. 00000100:00000001:3.0:1496184950.656528:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.656528:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 02000000:00000001:5.0:1496184950.656530:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.656532:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000010:3.0:1496184950.656532:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100e39c600. 00000020:00000001:5.0:1496184950.656533:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000010:3.0:1496184950.656535:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8e80. 00010000:00000080:5.0:1496184950.656536:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184950.656538:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2600. 00010000:00000040:5.0:1496184950.656541:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00000100:00000001:3.0:1496184950.656541:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00010000:00000001:5.0:1496184950.656542:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00000100:00000001:3.0:1496184950.656542:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00010000:00000001:5.0:1496184950.656543:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000100:00000001:3.0:1496184950.656543:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.656544:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.656545:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.656547:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.656548:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:5.0:1496184950.656553:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23939us (24086us total) trans 0 rc -110/-999 00000100:00000200:3.0:1496184950.656554:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b89850 x1568861104107632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:5.0:1496184950.656558:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57708 00000100:00000040:5.0:1496184950.656560:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.656560:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.656561:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.656561:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.656562:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000001:3.0:1496184950.656564:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00002000:00100000:5.0:1496184950.656565:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600ac50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00004000:00000001:3.0:1496184950.656568:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.656569:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:3.0:1496184950.656570:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00002000:00000001:5.0:1496184950.656571:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.656571:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600ac50 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184950.656571:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00010000:00000001:5.0:1496184950.656573:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.656574:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000100:00000001:3.0:1496184950.656574:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000020:00000001:5.0:1496184950.656575:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000100:00000001:3.0:1496184950.656576:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000001:5.0:1496184950.656577:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.656578:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.656579:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.656579:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00000100:00100000:3.0:1496184950.656579:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6113 00002000:00010000:5.0:1496184950.656580:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600ac50. 00002000:00000001:5.0:1496184950.656581:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.656581:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.656583:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000100:00000001:3.0:1496184950.656583:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736386128 : -131906973165488 : ffff880803b89850) 00000020:00000010:5.0:1496184950.656585:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8d80. 00000020:00000010:5.0:1496184950.656586:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000100:00000040:3.0:1496184950.656586:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b89850 x1568861104107632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000010:5.0:1496184950.656588:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.656589:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:5.0:1496184950.656591:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1496184950.656591:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.656592:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00000200:5.0:1496184950.656593:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00100000:3.0:1496184950.656594:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107632:12345-10.37.248.124@o2ib1:8 00000100:00000200:3.0:1496184950.656596:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107632 00004000:00000001:5.0:1496184950.656598:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.656599:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:00000001:3.0:1496184950.656599:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184950.656603:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184950.656611:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184950.656616:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184950.656618:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184950.656621:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fc400. 02000000:00000001:3.0:1496184950.656622:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656624:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.656630:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184950.656639:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184950.656642:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184950.656644:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184950.656646:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184950.656649:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:3.0:1496184950.656653:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.656656:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.656658:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.656660:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.656662:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.656664:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.656665:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.656667:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.656671:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.656672:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.656673:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.656675:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.656677:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.656682:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.656683:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.656685:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.656687:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.656688:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.656691:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.656693:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.656694:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.656695:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.656697:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.656699:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.656701:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.656702:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.656704:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.656706:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.656707:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.656709:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.656710:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.656711:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.656714:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.656715:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.656716:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.656718:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.656720:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.656722:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656724:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.656728:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.656730:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.656733:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.656735:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.656736:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.656738:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656739:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.656740:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.656742:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.656743:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.656746:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.656748:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656749:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.656752:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.656753:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.656754:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.656756:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.656757:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.656759:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.656761:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.656762:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.656764:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.656766:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.656769:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.656771:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.656773:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.656774:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.656775:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.656777:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.656778:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.656780:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.656782:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.656785:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107632 00010000:00000001:3.0:1496184950.656787:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.656790:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b89850 x1568861104107632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.656797:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.656798:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.656800:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.656803:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.656805:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.656806:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.656807:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.656809:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.656810:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656812:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.656814:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.656818:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107632, offset 0 00000100:00000001:3.0:1496184950.656842:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.656843:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.656845:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.656846:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.656847:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.656848:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.656849:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.656851:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b89850 x1568861104107632/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.656857:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107632:12345-10.37.248.124@o2ib1:8 Request procesed in 264us (384us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.656862:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6113 00000100:00000040:3.0:1496184950.656864:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.656866:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.656866:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.656870:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8e80. 00000020:00000010:3.0:1496184950.656872:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2600. 00000020:00000010:3.0:1496184950.656879:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100e39c600. 00000020:00000040:3.0:1496184950.656881:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.656883:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.656885:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.656900:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.656901:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.657223:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.657238:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.657240:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fc400. 02000000:00000001:6.0:1496184950.657243:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.657244:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.657635:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.657637:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.657639:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b850 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.657644:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.657645:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.657647:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.657680:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.657683:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.657685:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.657686:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.657687:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.657688:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.657691:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.657693:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.657694:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.657695:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.657697:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.657698:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.657699:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.657701:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.657704:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.657705:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.657706:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.657709:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8a00. 00000020:00000010:5.0:1496184950.657710:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.657712:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.657714:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.657715:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.657716:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.657717:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.657718:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.657719:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.657720:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.657722:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.657722:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.657724:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.657725:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.657726:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.657726:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.657727:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.657728:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657729:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.657730:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657731:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657732:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.657734:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.657735:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.657736:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.657739:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.657750:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.657751:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.657752:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.657753:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.657754:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.657755:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.657757:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.657758:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.657758:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.657759:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00002000:00000001:5.0:1496184950.657760:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.657761:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.657763:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.657768:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.657770:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.657771:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.657772:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.657773:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.657774:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.657775:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.657777:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57709 00000100:00000040:5.0:1496184950.657779:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.657780:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671276624 : -131872038274992 : ffff88102600b850) 00000100:00000040:5.0:1496184950.657783:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.657788:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.657788:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.657790:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.657792:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.657793:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.657795:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.657796:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657797:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.657798:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.657799:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.657801:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.657802:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.657802:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.657803:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657804:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.657806:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.657807:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.657808:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.657810:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.657811:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657812:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.657813:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.657815:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.657819:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.657822:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.657823:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.657825:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.657827:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.657828:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.657829:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.657830:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.657832:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.657833:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.657835:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.657837:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.657837:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.657838:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.657839:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.657841:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.657841:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.657842:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.657845:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.657867:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.657873:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657874:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.657877:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.657878:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.657879:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.657880:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.657881:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.657882:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.657884:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.657885:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.657902:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.657904:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.657931:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff080-0x592def0cff080 00000100:00000001:5.0:1496184950.657933:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.669930:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.669932:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.680905:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.680910:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.680911:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.691886:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.691955:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.691962:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.691964:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.691968:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.691969:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.691971:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.691973:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.692006:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.692008:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.692010:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.692011:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.692012:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.692013:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.692014:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.692015:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.692016:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.692017:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.692018:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.692020:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.692022:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.692023:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.692024:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.692025:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.692026:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.692028:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.692030:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.692032:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.692037:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.692039:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.692040:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.692040:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.692042:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.692043:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.692049:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34260us (34409us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.692054:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57709 00000100:00000040:5.0:1496184950.692056:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.692057:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.692058:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.692061:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.692066:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.692067:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.692069:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.692070:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.692071:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.692072:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.692074:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.692074:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.692075:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.692076:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b850. 00002000:00000001:5.0:1496184950.692077:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.692078:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.692080:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8a00. 00000020:00000010:5.0:1496184950.692081:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.692083:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.692084:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.692086:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.692088:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.692094:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.692095:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.702884:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.702885:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.702888:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810265c8050 x1568861104107664 msgsize 520 00000100:00100000:6.0:1496184950.702904:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.702905:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.702908:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.702940:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.702942:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.702944:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.702945:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.702946:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.702947:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.702949:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.702951:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107664 00000020:00000001:5.0:1496184950.702953:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.702954:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.702955:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.702956:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.702958:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.702959:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.702961:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.702962:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.702964:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.702966:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8780. 00000020:00000010:5.0:1496184950.702967:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.702969:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.702970:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.702971:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.702973:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.702976:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810265c8050 x1568861104107664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.702981:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.702982:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.702987:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.702989:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.702990:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.702990:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.702993:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.702994:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.702995:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.702997:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43435 00000100:00000040:5.0:1496184950.702998:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.703000:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201677291600 : -131872032260016 : ffff8810265c8050) 00000100:00000040:5.0:1496184950.703002:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810265c8050 x1568861104107664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.703007:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.703008:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.703009:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107664:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.703011:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107664 00000020:00000001:5.0:1496184950.703013:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.703015:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.703018:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.703021:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.703022:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.703023:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68e400. 02000000:00000001:5.0:1496184950.703024:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703025:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.703028:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.703030:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.703031:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.703033:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.703035:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.703037:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.703039:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.703041:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.703042:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.703043:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.703044:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.703045:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.703046:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.703048:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.703049:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.703050:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.703051:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.703052:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.703054:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.703057:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.703058:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.703059:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.703061:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.703062:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.703064:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.703065:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.703066:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.703067:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.703069:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.703071:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.703072:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703073:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.703074:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.703076:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.703078:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703079:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.703080:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.703081:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.703083:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.703084:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.703084:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.703086:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.703087:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.703089:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703090:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.703093:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.703095:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.703096:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.703098:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.703099:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.703101:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703102:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.703103:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.703104:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.703106:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.703107:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.703109:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703110:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.703111:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.703112:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.703113:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.703115:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.703115:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.703117:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.703119:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.703120:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.703121:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.703123:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703124:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.703125:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.703127:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.703128:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.703129:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.703129:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.703130:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703131:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.703133:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.703134:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107664 00010000:00000001:5.0:1496184950.703135:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.703137:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810265c8050 x1568861104107664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.703142:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.703143:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.703144:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.703147:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.703148:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.703150:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.703151:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.703152:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.703153:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703154:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.703156:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.703158:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107664, offset 0 00000100:00000001:5.0:1496184950.703172:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703173:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.703175:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.703176:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.703177:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.703178:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703179:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.703181:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810265c8050 x1568861104107664/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.703186:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107664:12345-10.37.248.124@o2ib1:8 Request procesed in 178us (297us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.703191:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43435 00000100:00000040:5.0:1496184950.703192:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.703194:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.703194:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.703196:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8780. 00000020:00000010:5.0:1496184950.703197:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.703199:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.703200:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.703202:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.703203:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.703207:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.703208:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.703257:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.703259:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.703261:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68e400. 02000000:00000001:6.0:1496184950.703264:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.703264:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.703732:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.703733:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.703736:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600b450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.703741:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.703742:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.703744:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.703776:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.703779:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.703780:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.703781:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.703783:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.703784:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.703786:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.703788:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.703789:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.703790:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.703791:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.703793:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.703794:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.703796:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.703798:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.703799:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.703801:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.703803:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8700. 00000020:00000010:5.0:1496184950.703805:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.703807:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.703808:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.703809:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.703810:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.703811:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.703812:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.703814:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.703815:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.703816:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.703817:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.703818:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.703819:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.703820:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.703821:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.703822:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.703822:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703823:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.703825:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703826:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703827:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703828:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.703829:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703830:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.703833:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.703839:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.703840:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.703842:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.703842:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.703843:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.703845:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.703846:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.703847:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.703848:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.703848:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00002000:00000001:5.0:1496184950.703849:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703851:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703853:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703857:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.703859:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.703860:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.703861:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.703862:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.703863:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.703864:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.703866:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57710 00000100:00000040:5.0:1496184950.703868:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.703869:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671275600 : -131872038276016 : ffff88102600b450) 00000100:00000040:5.0:1496184950.703871:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.703876:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.703877:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.703879:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.703881:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.703882:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.703884:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.703884:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703885:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.703886:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.703888:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.703889:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.703902:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.703903:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.703903:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703904:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.703906:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.703907:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.703909:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.703910:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703911:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703912:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.703913:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.703915:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.703920:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.703922:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.703923:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.703925:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.703927:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.703928:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.703929:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.703930:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.703932:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.703933:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.703935:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.703936:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.703937:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.703937:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.703938:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.703940:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.703941:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.703942:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.703944:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.703966:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.703972:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703973:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.703976:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703977:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.703978:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.703979:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.703989:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.703990:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.703991:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.703993:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.703997:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.703998:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.704028:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff0a0-0x592def0cff0a0 00000100:00000001:5.0:1496184950.704030:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.705662:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.705664:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.705670:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.705675:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.705675:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.705680:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.705717:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.705724:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.705726:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.705730:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.705731:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.705733:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.705735:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.705768:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.705770:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.705772:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.705773:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.705773:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.705775:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.705776:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.705777:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.705778:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.705779:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.705780:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.705781:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.705783:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.705784:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.705785:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.705786:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.705788:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.705789:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.705791:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.705794:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.705799:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.705800:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.705801:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.705802:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.705803:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.705805:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.705810:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1933us (2073us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.705815:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57710 00000100:00000040:5.0:1496184950.705817:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.705818:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.705819:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.705822:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600b450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.705827:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.705828:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600b450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.705830:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.705830:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.705832:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.705833:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.705835:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.705835:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.705836:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.705836:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600b450. 00002000:00000001:5.0:1496184950.705838:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.705839:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.705841:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8700. 00000020:00000010:5.0:1496184950.705842:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.705844:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.705845:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.705847:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.705849:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.705854:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.705855:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.734499:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.734501:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.734504:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7450 x1568861104107696 msgsize 520 00000100:00100000:7.0:1496184950.734509:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.734510:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.734512:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.734543:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.734546:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.734548:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.734549:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.734550:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.734551:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.734553:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.734555:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107696 00000020:00000001:5.0:1496184950.734557:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.734557:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.734559:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.734560:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.734561:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.734563:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.734565:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.734566:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.734568:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.734569:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8f80. 00000020:00000010:5.0:1496184950.734571:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.734573:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.734573:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.734574:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.734576:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.734580:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7450 x1568861104107696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.734585:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.734585:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734590:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.734592:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.734593:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.734594:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.734595:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.734595:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.734596:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.734599:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43436 00000100:00000040:5.0:1496184950.734600:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.734601:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676239952 : -131872033311664 : ffff8810264c7450) 00000100:00000040:5.0:1496184950.734604:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7450 x1568861104107696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.734609:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.734609:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.734611:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107696:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.734613:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107696 00000020:00000001:5.0:1496184950.734614:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.734616:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.734619:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.734622:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.734623:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.734624:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68f800. 02000000:00000001:5.0:1496184950.734626:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734627:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.734629:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.734631:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.734633:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.734634:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.734636:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.734638:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.734640:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.734641:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.734642:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.734643:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.734644:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.734645:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.734646:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.734648:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.734649:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.734650:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.734651:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.734652:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.734653:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.734656:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.734657:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.734658:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.734660:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.734661:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.734663:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.734664:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.734665:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.734666:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.734668:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.734669:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.734671:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.734672:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.734673:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.734674:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.734676:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.734677:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.734678:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.734679:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.734680:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.734681:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.734682:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.734684:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.734685:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.734686:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734688:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.734690:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.734692:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.734693:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.734695:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.734696:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.734697:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734699:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.734700:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.734701:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.734703:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.734704:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.734705:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734706:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.734707:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.734708:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.734709:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.734710:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.734711:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.734712:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.734714:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.734715:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.734716:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.734717:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.734719:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.734720:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.734722:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.734722:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.734723:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.734724:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.734725:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.734726:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.734727:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.734728:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107696 00010000:00000001:5.0:1496184950.734730:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.734732:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7450 x1568861104107696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.734737:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.734738:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.734739:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.734741:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.734743:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.734744:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.734746:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.734747:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.734748:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734749:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.734750:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.734752:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107696, offset 0 00000100:00000001:5.0:1496184950.734763:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.734764:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.734765:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.734766:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.734767:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.734768:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.734769:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.734771:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7450 x1568861104107696/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.734777:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107696:12345-10.37.248.124@o2ib1:8 Request procesed in 166us (271us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.734781:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43436 00000100:00000040:5.0:1496184950.734782:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.734784:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.734784:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.734786:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8f80. 00000020:00000010:5.0:1496184950.734788:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.734789:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.734790:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.734792:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.734794:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.734798:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.734799:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.734847:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.734849:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.734851:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68f800. 02000000:00000001:7.0:1496184950.734854:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.734855:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.735329:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.735330:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.735333:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c050 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.735338:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.735339:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.735341:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.735375:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.735378:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.735379:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.735380:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.735381:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.735382:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.735385:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.735386:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.735388:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.735389:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.735390:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.735391:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.735393:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.735394:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.735396:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.735397:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.735399:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.735401:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8400. 00000020:00000010:5.0:1496184950.735402:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.735404:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.735406:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.735406:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.735407:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.735408:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.735409:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.735411:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.735412:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.735413:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.735414:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.735415:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.735416:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.735417:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.735418:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.735419:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.735420:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735421:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.735422:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735423:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735424:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.735425:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.735426:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.735427:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.735430:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.735437:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.735437:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.735439:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.735440:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.735441:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.735442:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.735444:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.735445:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.735445:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.735446:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00002000:00000001:5.0:1496184950.735447:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.735448:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.735450:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.735455:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.735457:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.735458:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.735459:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.735459:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.735460:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.735461:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.735464:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57711 00000100:00000040:5.0:1496184950.735466:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.735467:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671278672 : -131872038272944 : ffff88102600c050) 00000100:00000040:5.0:1496184950.735469:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.735474:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.735475:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.735477:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.735479:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.735480:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.735481:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.735482:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735483:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.735484:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.735485:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.735486:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.735487:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.735488:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.735489:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735490:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.735492:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.735492:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.735494:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68d800. 02000000:00000001:5.0:1496184950.735495:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.735496:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735497:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.735498:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.735500:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.735504:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.735507:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.735508:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.735510:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.735512:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.735513:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.735513:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.735515:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.735517:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.735517:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.735519:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.735521:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.735521:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.735522:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.735523:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.735525:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.735525:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.735526:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.735529:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.735552:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.735558:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735559:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.735562:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.735563:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.735564:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.735565:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.735566:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.735567:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.735569:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.735570:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.735575:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.735576:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.735602:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff0c0-0x592def0cff0c0 00000100:00000001:5.0:1496184950.735604:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.737149:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.737152:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.737161:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.737179:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.737179:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.737183:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.737222:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.737237:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.737239:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.737243:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.737244:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.737246:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.737248:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.737281:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.737284:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.737285:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.737286:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.737287:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.737288:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.737289:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.737291:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.737291:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.737292:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.737293:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.737295:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.737297:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.737298:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.737299:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.737300:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68d800. 02000000:00000001:5.0:1496184950.737301:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.737303:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.737305:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.737307:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.737312:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.737314:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.737314:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.737315:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.737316:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.737318:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.737324:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1849us (1990us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.737329:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57711 00000100:00000040:5.0:1496184950.737331:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.737332:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.737333:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.737336:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.737341:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.737342:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.737343:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.737344:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.737345:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.737347:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.737348:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.737349:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.737350:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.737350:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c050. 00002000:00000001:5.0:1496184950.737352:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.737353:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.737354:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8400. 00000020:00000010:5.0:1496184950.737356:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.737357:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.737359:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.737360:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.737363:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.737368:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.737369:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.765484:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.765485:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.765488:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7c50 x1568861104107728 msgsize 520 00000100:00100000:7.0:1496184950.765493:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.765494:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.765496:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.765526:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.765528:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.765530:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.765531:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.765533:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.765534:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.765536:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.765538:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107728 00000020:00000001:5.0:1496184950.765539:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.765540:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.765541:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.765542:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.765544:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.765546:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.765548:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.765548:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.765550:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.765552:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8a80. 00000020:00000010:5.0:1496184950.765554:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.765555:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.765556:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.765557:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.765559:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.765562:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7c50 x1568861104107728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.765567:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.765568:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765573:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.765575:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.765575:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.765576:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.765577:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.765578:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.765579:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.765581:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43437 00000100:00000040:5.0:1496184950.765583:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.765584:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676242000 : -131872033309616 : ffff8810264c7c50) 00000100:00000040:5.0:1496184950.765586:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7c50 x1568861104107728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.765591:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.765592:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.765594:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107728:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.765596:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107728 00000020:00000001:5.0:1496184950.765597:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.765599:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.765602:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.765604:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.765605:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.765607:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68d800. 02000000:00000001:5.0:1496184950.765608:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765609:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.765612:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.765613:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.765615:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.765616:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.765618:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.765620:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.765622:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.765624:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.765624:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.765625:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.765626:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.765627:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.765628:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.765630:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.765631:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.765632:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.765633:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.765634:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.765635:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.765639:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.765640:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.765641:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.765643:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.765643:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.765645:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.765647:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.765647:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.765648:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.765650:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.765652:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.765653:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.765654:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.765655:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.765657:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.765658:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.765659:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.765660:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.765661:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.765663:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.765664:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.765665:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.765666:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.765667:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.765669:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765670:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.765672:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.765674:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.765676:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.765678:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.765679:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.765680:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765681:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.765682:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.765683:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.765685:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.765686:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.765687:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765689:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.765690:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.765690:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.765691:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.765693:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.765694:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.765695:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.765696:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.765698:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.765699:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.765700:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.765701:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.765702:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.765704:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.765705:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.765706:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.765706:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.765707:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.765708:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.765710:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.765711:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107728 00010000:00000001:5.0:1496184950.765712:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.765714:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7c50 x1568861104107728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.765719:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.765720:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.765722:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.765724:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.765725:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.765727:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.765728:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.765729:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.765730:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765731:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.765733:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.765735:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107728, offset 0 00000100:00000001:5.0:1496184950.765746:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.765747:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.765748:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.765749:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.765750:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.765751:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.765752:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.765754:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7c50 x1568861104107728/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.765759:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107728:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (270us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.765764:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43437 00000100:00000040:5.0:1496184950.765765:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.765766:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.765767:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.765769:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8a80. 00000020:00000010:5.0:1496184950.765770:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.765772:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.765773:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.765775:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.765776:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.765780:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.765781:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.765835:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.765837:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.765838:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68d800. 02000000:00000001:6.0:1496184950.765841:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.765842:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.766273:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.766275:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.766277:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.766282:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.766283:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.766285:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.766317:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.766320:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.766321:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.766322:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.766324:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.766325:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.766327:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.766329:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.766330:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.766331:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.766332:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.766334:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.766335:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.766337:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.766339:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.766340:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.766341:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.766343:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff8808076a8e00. 00000020:00000010:5.0:1496184950.766345:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.766347:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.766348:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.766349:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.766350:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.766351:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.766352:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.766354:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.766355:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.766356:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.766357:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.766358:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.766359:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.766360:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.766361:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.766362:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.766362:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766363:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.766365:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766366:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766367:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.766368:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.766369:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.766370:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.766373:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.766379:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.766380:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.766382:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.766383:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.766383:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.766385:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.766386:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.766387:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.766388:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.766388:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00002000:00000001:5.0:1496184950.766389:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.766391:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.766393:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.766397:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.766399:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.766400:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.766401:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.766402:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.766403:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.766403:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.766406:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57712 00000100:00000040:5.0:1496184950.766407:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.766409:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671279696 : -131872038271920 : ffff88102600c450) 00000100:00000040:5.0:1496184950.766411:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.766416:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.766417:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.766419:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.766421:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.766422:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.766423:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.766424:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766425:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.766426:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.766427:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.766428:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.766429:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.766430:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.766431:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766432:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.766434:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.766434:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.766436:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68dc00. 02000000:00000001:5.0:1496184950.766437:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.766438:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766440:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.766440:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.766442:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.766446:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.766449:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.766450:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.766452:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.766454:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.766455:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.766456:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.766457:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.766459:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.766460:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.766462:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.766463:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.766464:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.766464:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.766465:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.766467:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.766468:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.766469:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.766471:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.766493:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.766498:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766500:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.766502:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.766504:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.766505:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.766506:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.766507:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.766508:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.766509:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.766511:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.766515:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.766516:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.766542:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff0e0-0x592def0cff0e0 00000100:00000001:5.0:1496184950.766545:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.767920:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.767922:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.767928:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.767932:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.767933:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.767938:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.767985:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.767991:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.767994:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.767997:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.767998:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.768001:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.768002:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.768035:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.768038:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.768040:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.768041:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.768041:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.768043:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.768044:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.768045:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.768046:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.768047:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.768048:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.768049:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.768051:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.768052:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.768053:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.768054:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68dc00. 02000000:00000001:5.0:1496184950.768056:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.768057:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.768059:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.768062:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.768067:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.768068:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.768069:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.768070:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.768071:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.768073:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.768079:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1661us (1800us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.768083:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57712 00000100:00000040:5.0:1496184950.768085:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.768086:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.768087:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.768090:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.768095:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.768096:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.768098:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.768099:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.768100:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.768101:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.768103:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.768103:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.768104:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.768105:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c450. 00002000:00000001:5.0:1496184950.768106:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.768107:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.768109:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff8808076a8e00. 00000020:00000010:5.0:1496184950.768110:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.768112:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.768113:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.768115:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.768117:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.768122:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.768123:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.796302:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.796304:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.796306:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6c50 x1568861104107760 msgsize 520 00000100:00100000:7.0:1496184950.796311:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.796312:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.796315:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.796346:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.796348:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.796350:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.796351:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.796352:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.796353:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.796356:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.796357:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107760 00000020:00000001:5.0:1496184950.796359:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.796360:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.796361:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.796362:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.796364:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.796365:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.796368:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.796368:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.796370:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.796372:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88085de54280. 00000020:00000010:5.0:1496184950.796374:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.796376:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.796376:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.796377:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.796379:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.796383:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6c50 x1568861104107760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.796388:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.796388:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796393:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.796395:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.796396:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.796397:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.796397:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.796398:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.796399:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.796401:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43438 00000100:00000040:5.0:1496184950.796403:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.796404:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676237904 : -131872033313712 : ffff8810264c6c50) 00000100:00000040:5.0:1496184950.796406:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6c50 x1568861104107760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.796411:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.796412:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.796414:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107760:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.796416:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107760 00000020:00000001:5.0:1496184950.796417:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.796419:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.796422:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.796425:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.796426:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.796427:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68dc00. 02000000:00000001:5.0:1496184950.796428:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796429:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.796432:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.796433:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.796435:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.796436:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.796438:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.796440:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.796443:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.796444:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.796445:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.796446:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.796447:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.796447:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.796449:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.796450:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.796451:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.796452:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.796453:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.796454:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.796455:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.796459:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.796459:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.796461:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.796462:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.796463:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.796465:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.796467:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.796467:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.796469:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.796470:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.796472:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.796473:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.796474:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.796475:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.796477:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.796478:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.796479:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.796480:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.796482:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.796483:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.796484:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.796485:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.796486:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.796488:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.796489:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796490:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.796492:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.796495:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.796496:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.796498:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.796499:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.796500:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796501:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.796502:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.796503:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.796505:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.796506:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.796507:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796508:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.796509:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.796510:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.796511:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.796513:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.796513:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.796515:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.796516:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.796517:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.796519:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.796520:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.796521:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.796522:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.796524:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.796525:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.796526:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.796526:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.796527:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.796528:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.796530:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.796531:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107760 00010000:00000001:5.0:1496184950.796532:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.796534:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6c50 x1568861104107760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.796539:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.796540:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.796541:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.796544:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.796545:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.796547:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.796548:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.796549:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.796550:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796551:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.796553:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.796555:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107760, offset 0 00000100:00000001:5.0:1496184950.796566:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.796567:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.796568:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.796569:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.796570:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.796571:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.796572:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.796574:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6c50 x1568861104107760/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.796579:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107760:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (271us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.796584:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43438 00000100:00000040:5.0:1496184950.796585:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.796587:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.796587:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.796589:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88085de54280. 00000020:00000010:5.0:1496184950.796591:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.796592:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.796593:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.796595:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.796597:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.796600:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.796601:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.796669:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.796672:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.796674:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68dc00. 02000000:00000001:7.0:1496184950.796679:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.796680:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.797162:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.797164:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.797166:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600bc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.797171:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.797181:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.797183:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.797216:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.797218:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.797220:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.797221:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.797222:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.797223:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.797226:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.797228:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.797229:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.797230:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.797231:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.797232:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.797234:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.797236:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.797238:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.797239:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.797240:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.797243:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb980. 00000020:00000010:5.0:1496184950.797244:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.797246:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.797248:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.797248:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.797249:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.797250:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.797251:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.797253:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.797254:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.797255:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.797256:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.797257:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.797258:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.797259:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.797260:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.797261:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.797261:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797262:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.797264:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797265:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797266:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.797267:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.797268:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.797269:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.797272:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.797278:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.797279:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.797281:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.797282:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.797283:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.797284:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.797286:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.797286:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.797287:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.797287:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00002000:00000001:5.0:1496184950.797289:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.797290:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.797292:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.797296:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.797298:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.797299:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.797300:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.797301:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.797302:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.797303:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.797305:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57713 00000100:00000040:5.0:1496184950.797306:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.797308:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671277648 : -131872038273968 : ffff88102600bc50) 00000100:00000040:5.0:1496184950.797310:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.797315:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.797316:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.797317:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.797319:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.797321:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.797322:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.797323:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797324:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.797325:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.797326:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.797327:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.797328:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.797329:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.797330:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797331:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.797333:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.797334:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.797335:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68b000. 02000000:00000001:5.0:1496184950.797336:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.797337:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797339:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.797339:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.797341:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.797345:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.797348:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.797349:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.797351:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.797353:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.797354:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.797355:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.797356:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.797358:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.797359:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.797361:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.797362:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.797363:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.797363:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.797364:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.797366:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.797367:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.797368:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.797370:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.797393:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.797399:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797400:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.797403:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.797404:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.797405:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.797406:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.797407:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.797409:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.797410:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.797411:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.797416:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.797417:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.797443:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff100-0x592def0cff100 00000100:00000001:5.0:1496184950.797445:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.799031:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.799034:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.810146:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.810150:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.810151:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.821129:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.821208:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.821215:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.821217:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.821221:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.821222:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.821224:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.821225:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.821258:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.821261:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.821263:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.821264:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.821264:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.821266:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.821267:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.821268:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.821269:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.821269:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.821271:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.821272:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.821274:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.821276:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.821277:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.821278:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68b000. 02000000:00000001:5.0:1496184950.821279:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.821281:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.821282:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.821285:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.821290:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.821291:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.821292:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.821293:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.821294:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.821296:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.821302:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23985us (24134us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.821307:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57713 00000100:00000040:5.0:1496184950.821308:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.821309:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.821310:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.821313:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600bc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.821318:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.821319:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600bc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.821321:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.821321:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.821323:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.821324:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.821326:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.821326:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.821327:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.821328:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600bc50. 00002000:00000001:5.0:1496184950.821329:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.821330:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.821331:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb980. 00000020:00000010:5.0:1496184950.821333:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.821335:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.821336:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.821337:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.821340:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.821345:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.821346:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.831863:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.831865:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.831868:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7050 x1568861104107792 msgsize 520 00000100:00100000:6.0:1496184950.831873:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.831874:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.831876:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.831910:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.831913:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.831914:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.831915:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.831916:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.831917:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.831920:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.831922:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107792 00000020:00000001:5.0:1496184950.831923:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.831924:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.831925:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.831926:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.831928:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.831929:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.831932:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.831932:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.831934:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.831936:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb180. 00000020:00000010:5.0:1496184950.831937:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.831939:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.831940:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.831941:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.831943:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.831946:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7050 x1568861104107792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.831951:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.831952:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.831957:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.831959:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.831959:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.831960:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.831961:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.831962:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.831963:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.831965:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43439 00000100:00000040:5.0:1496184950.831966:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.831968:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676238928 : -131872033312688 : ffff8810264c7050) 00000100:00000040:5.0:1496184950.831970:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7050 x1568861104107792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.831975:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.831976:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.831977:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107792:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.831980:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107792 00000020:00000001:5.0:1496184950.831981:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.831983:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.831986:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.831988:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.831989:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.831991:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68b000. 02000000:00000001:5.0:1496184950.831992:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.831993:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.831996:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.831997:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.831999:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.832000:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.832002:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.832004:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.832006:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.832008:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.832009:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.832010:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.832011:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.832011:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.832012:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.832014:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.832015:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.832016:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.832017:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.832018:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.832019:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.832023:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.832023:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.832025:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.832027:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.832027:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.832029:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.832031:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.832031:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.832032:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.832034:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.832036:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.832037:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832038:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.832039:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.832041:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.832042:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832043:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.832044:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.832046:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.832047:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.832048:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.832049:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.832050:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.832052:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.832053:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832054:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.832056:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.832059:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.832060:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.832062:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.832063:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.832064:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832065:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.832066:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.832068:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.832069:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.832070:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.832072:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832073:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.832074:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.832075:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.832076:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.832077:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.832078:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.832079:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.832081:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.832082:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.832083:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.832084:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832086:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.832087:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.832088:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.832089:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.832090:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.832091:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.832092:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832093:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.832094:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.832095:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107792 00010000:00000001:5.0:1496184950.832097:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.832098:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7050 x1568861104107792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.832104:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.832105:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.832106:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.832108:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.832110:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.832111:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.832112:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.832113:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.832114:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832115:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.832117:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.832119:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107792, offset 0 00000100:00000001:5.0:1496184950.832130:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832131:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.832132:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.832133:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.832134:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.832135:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832136:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.832137:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7050 x1568861104107792/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.832147:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107792:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (277us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.832151:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43439 00000100:00000040:5.0:1496184950.832152:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.832154:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.832155:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.832156:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb180. 00000020:00000010:5.0:1496184950.832158:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.832159:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.832160:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.832162:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.832164:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.832167:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.832168:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.832214:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.832216:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.832218:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68b000. 02000000:00000001:6.0:1496184950.832221:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.832222:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.832661:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.832663:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.832665:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.832670:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.832671:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.832677:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.832706:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.832709:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.832710:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.832711:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.832712:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.832713:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.832716:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.832718:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.832719:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.832720:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.832721:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.832722:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.832724:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.832726:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.832728:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.832728:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.832730:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.832732:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb500. 00000020:00000010:5.0:1496184950.832734:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.832736:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.832738:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.832739:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.832740:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.832741:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.832742:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.832743:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.832744:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.832746:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.832746:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.832748:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.832748:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.832750:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.832750:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.832751:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.832752:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832753:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.832754:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832755:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832756:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832757:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.832758:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832760:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.832762:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.832769:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.832770:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.832771:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.832772:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.832773:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.832775:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.832776:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.832777:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.832777:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.832778:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00002000:00000001:5.0:1496184950.832779:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832780:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832783:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832787:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.832789:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.832790:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.832791:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.832792:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.832793:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.832794:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.832796:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57714 00000100:00000040:5.0:1496184950.832798:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.832799:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671282768 : -131872038268848 : ffff88102600d050) 00000100:00000040:5.0:1496184950.832801:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.832806:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.832807:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.832809:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.832811:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.832812:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.832814:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.832815:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832816:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.832816:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.832818:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.832819:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.832820:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.832820:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.832821:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832822:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.832824:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.832825:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.832826:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184950.832827:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832829:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832830:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.832831:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.832833:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.832837:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.832839:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.832840:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.832843:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.832844:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.832845:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.832846:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.832848:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.832849:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.832850:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.832852:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.832854:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.832854:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.832855:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.832856:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.832858:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.832858:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.832859:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.832862:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.832884:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.832905:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832907:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.832910:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832911:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.832912:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.832913:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.832914:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.832916:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.832917:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.832919:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.832923:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.832924:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.832949:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff120-0x592def0cff120 00000100:00000001:5.0:1496184950.832952:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.834398:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.834401:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.845445:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.845450:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.845451:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.856481:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.856544:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.856563:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.856566:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00000100:00000001:1.0:1496184950.856567:0:21159:0:(events.c:294:request_in_callback()) Process entered 00002000:00000001:5.0:1496184950.856569:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00000100:00000200:1.0:1496184950.856569:0:21159:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00002000:00000001:5.0:1496184950.856570:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00000100:00000040:1.0:1496184950.856572:0:21159:0:(events.c:347:request_in_callback()) incoming req@ffff880803b88050 x1568861104107824 msgsize 520 00002000:00000001:5.0:1496184950.856573:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.856574:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000100:00100000:1.0:1496184950.856579:0:21159:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:1.0:1496184950.856580:0:21159:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:1.0:1496184950.856583:0:21159:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:3.0:1496184950.856589:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:3.0:1496184950.856592:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:3.0:1496184950.856596:0:17421:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:3.0:1496184950.856598:0:17421:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:3.0:1496184950.856601:0:17421:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:3.0:1496184950.856602:0:17421:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.856607:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1496184950.856607:0:17421:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.856610:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00000100:00100000:3.0:1496184950.856610:0:17421:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107824 00002000:00000001:5.0:1496184950.856612:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.856613:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.856613:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:3.0:1496184950.856613:0:17421:0:(genops.c:722:class_conn2export()) Process entered 00002000:00000001:5.0:1496184950.856615:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000040:3.0:1496184950.856615:0:17421:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.856616:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.856617:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:3.0:1496184950.856617:0:17421:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.856618:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.856618:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.856620:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.856620:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 10 00000100:00000010:5.0:1496184950.856621:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000001:3.0:1496184950.856622:0:17421:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000010:5.0:1496184950.856623:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000020:00000001:3.0:1496184950.856623:0:17421:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.856624:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.856625:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.856626:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184950.856628:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:3.0:1496184950.856628:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000020:02000400:5.0:1496184950.856629:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000100:00000001:3.0:1496184950.856629:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000001:5.0:1496184950.856631:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000010:3.0:1496184950.856631:0:17421:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100fcd2600. 00010000:00000080:5.0:1496184950.856634:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000020:00000010:3.0:1496184950.856635:0:17421:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100a6c8880. 00000020:00000010:3.0:1496184950.856638:0:17421:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff8810371d2600. 00010000:00000040:5.0:1496184950.856639:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.856641:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.856641:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000100:00000001:3.0:1496184950.856641:0:17421:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000020:00000001:5.0:1496184950.856642:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856642:0:17421:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.856643:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000001:3.0:1496184950.856643:0:17421:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:5.0:1496184950.856645:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.856648:0:17421:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00100000:5.0:1496184950.856651:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+9:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 23843us (23984us total) trans 0 rc -110/-999 00000100:00000200:3.0:1496184950.856654:0:17421:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff880803b88050 x1568861104107824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00100000:5.0:1496184950.856656:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57714 00000100:00000040:5.0:1496184950.856658:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.856659:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.856659:0:17421:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00002000:00000001:5.0:1496184950.856660:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00000100:00000001:3.0:1496184950.856661:0:17421:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.856663:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00000100:00000001:3.0:1496184950.856665:0:17421:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00002000:00000001:5.0:1496184950.856668:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00004000:00000001:3.0:1496184950.856668:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00002000:00100000:5.0:1496184950.856669:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d050 with x1568861102817424 ext(0->1048575) 00004000:00000001:3.0:1496184950.856670:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00010000:00000001:5.0:1496184950.856671:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00004000:00000001:3.0:1496184950.856671:0:17421:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00000020:00000001:5.0:1496184950.856672:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00004000:00000001:3.0:1496184950.856672:0:17421:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000020:00000001:5.0:1496184950.856673:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.856675:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00000100:00000001:3.0:1496184950.856675:0:17421:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00010000:00000001:5.0:1496184950.856676:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00000100:00000001:3.0:1496184950.856676:0:17421:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00010000:00000001:5.0:1496184950.856677:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.856677:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.856678:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d050. 00002000:00000001:5.0:1496184950.856679:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:3.0:1496184950.856680:0:17421:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 6114 00000100:00000001:5.0:1496184950.856681:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.856682:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb500. 00000100:00000040:3.0:1496184950.856682:0:17421:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.856683:0:17421:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612166736379984 : -131906973171632 : ffff880803b88050) 00000020:00000010:5.0:1496184950.856684:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.856686:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000100:00000040:3.0:1496184950.856686:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff880803b88050 x1568861104107824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000020:00000040:5.0:1496184950.856688:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000001:5.0:1496184950.856689:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.856691:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00000100:00000001:3.0:1496184950.856692:0:17421:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:3.0:1496184950.856693:0:17421:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:3.0:1496184950.856695:0:17421:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107824:12345-10.37.248.124@o2ib1:8 00004000:00000001:5.0:1496184950.856697:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00000100:00000200:3.0:1496184950.856697:0:17421:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107824 00004000:00000001:5.0:1496184950.856698:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000020:00000001:3.0:1496184950.856700:0:17421:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:3.0:1496184950.856704:0:17421:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:3.0:1496184950.856711:0:17421:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:3.0:1496184950.856716:0:17421:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:3.0:1496184950.856718:0:17421:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:3.0:1496184950.856720:0:17421:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff8807f23fc800. 02000000:00000001:3.0:1496184950.856722:0:17421:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856724:0:17421:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.856730:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:3.0:1496184950.856733:0:17421:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:3.0:1496184950.856736:0:17421:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:3.0:1496184950.856738:0:17421:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:3.0:1496184950.856747:0:17421:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:3.0:1496184950.856750:0:17421:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:3.0:1496184950.856754:0:17421:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:3.0:1496184950.856757:0:17421:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:3.0:1496184950.856759:0:17421:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:3.0:1496184950.856761:0:17421:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:3.0:1496184950.856763:0:17421:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:3.0:1496184950.856765:0:17421:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:3.0:1496184950.856767:0:17421:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:3.0:1496184950.856768:0:17421:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:3.0:1496184950.856771:0:17421:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:3.0:1496184950.856773:0:17421:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:3.0:1496184950.856775:0:17421:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:3.0:1496184950.856776:0:17421:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:3.0:1496184950.856778:0:17421:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:3.0:1496184950.856783:0:17421:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:3.0:1496184950.856784:0:17421:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:3.0:1496184950.856786:0:17421:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:3.0:1496184950.856788:0:17421:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:3.0:1496184950.856789:0:17421:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:3.0:1496184950.856792:0:17421:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:3.0:1496184950.856794:0:17421:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:3.0:1496184950.856795:0:17421:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:3.0:1496184950.856796:0:17421:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:3.0:1496184950.856798:0:17421:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:3.0:1496184950.856800:0:17421:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:3.0:1496184950.856802:0:17421:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.856803:0:17421:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:3.0:1496184950.856804:0:17421:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:3.0:1496184950.856807:0:17421:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:3.0:1496184950.856808:0:17421:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:3.0:1496184950.856809:0:17421:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.856810:0:17421:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:3.0:1496184950.856812:0:17421:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:3.0:1496184950.856814:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.856816:0:17421:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:3.0:1496184950.856817:0:17421:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:3.0:1496184950.856819:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.856821:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.856823:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856825:0:17421:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:3.0:1496184950.856828:0:17421:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:3.0:1496184950.856831:0:17421:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:3.0:1496184950.856833:0:17421:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:3.0:1496184950.856836:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.856837:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.856838:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856840:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.856841:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.856842:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.856844:0:17421:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:3.0:1496184950.856847:0:17421:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:3.0:1496184950.856848:0:17421:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856850:0:17421:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:3.0:1496184950.856852:0:17421:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:3.0:1496184950.856854:0:17421:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:3.0:1496184950.856855:0:17421:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:3.0:1496184950.856857:0:17421:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:3.0:1496184950.856858:0:17421:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:3.0:1496184950.856860:0:17421:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:3.0:1496184950.856862:0:17421:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:3.0:1496184950.856863:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:3.0:1496184950.856865:0:17421:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:3.0:1496184950.856867:0:17421:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.856870:0:17421:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:3.0:1496184950.856872:0:17421:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:3.0:1496184950.856874:0:17421:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:3.0:1496184950.856875:0:17421:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:3.0:1496184950.856876:0:17421:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:3.0:1496184950.856878:0:17421:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:3.0:1496184950.856879:0:17421:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.856881:0:17421:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:3.0:1496184950.856883:0:17421:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:3.0:1496184950.856886:0:17421:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107824 00010000:00000001:3.0:1496184950.856888:0:17421:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:3.0:1496184950.856906:0:17421:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff880803b88050 x1568861104107824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:3.0:1496184950.856913:0:17421:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:3.0:1496184950.856914:0:17421:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:3.0:1496184950.856916:0:17421:0:(import.c:1678:at_measured()) add 1 to ffff8807fc287d50 time=104 v=1 (1 1 1 1) 00000100:00001000:3.0:1496184950.856919:0:17421:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:3.0:1496184950.856921:0:17421:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:3.0:1496184950.856922:0:17421:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.856924:0:17421:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:3.0:1496184950.856925:0:17421:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:3.0:1496184950.856926:0:17421:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856928:0:17421:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:3.0:1496184950.856930:0:17421:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:3.0:1496184950.856933:0:17421:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107824, offset 0 00000100:00000001:3.0:1496184950.856949:0:17421:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.856950:0:17421:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:3.0:1496184950.856952:0:17421:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:3.0:1496184950.856953:0:17421:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:3.0:1496184950.856954:0:17421:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:3.0:1496184950.856955:0:17421:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1496184950.856956:0:17421:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:3.0:1496184950.856958:0:17421:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff880803b88050 x1568861104107824/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:3.0:1496184950.856964:0:17421:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_004:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107824:12345-10.37.248.124@o2ib1:8 Request procesed in 271us (390us total) trans 0 rc 0/0 00000100:00100000:3.0:1496184950.856969:0:17421:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 6114 00000100:00000040:3.0:1496184950.856971:0:17421:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:3.0:1496184950.856972:0:17421:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:3.0:1496184950.856973:0:17421:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:3.0:1496184950.856976:0:17421:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100a6c8880. 00000020:00000010:3.0:1496184950.856978:0:17421:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff8810371d2600. 00000020:00000010:3.0:1496184950.856981:0:17421:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100fcd2600. 00000020:00000040:3.0:1496184950.856983:0:17421:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:3.0:1496184950.856985:0:17421:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1496184950.856987:0:17421:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:3.0:1496184950.856993:0:17421:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:3.0:1496184950.856994:0:17421:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.857302:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.857316:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.857318:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff8807f23fc800. 02000000:00000001:7.0:1496184950.857329:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.857330:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.857690:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.857692:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.857695:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600c850 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.857700:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.857701:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.857704:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.857736:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.857738:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.857740:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.857741:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.857743:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.857744:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.857746:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.857748:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.857750:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.857751:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.857752:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.857753:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.857755:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.857756:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.857759:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.857768:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.857770:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.857772:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb680. 00000020:00000010:5.0:1496184950.857774:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.857776:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.857778:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.857778:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.857779:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.857781:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.857782:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.857783:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.857784:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.857786:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.857786:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.857788:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.857789:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.857790:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.857791:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.857792:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.857792:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857793:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.857794:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857795:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857797:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.857798:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.857799:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.857800:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.857803:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.857809:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.857810:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.857812:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.857812:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.857813:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.857815:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.857816:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.857817:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.857817:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.857818:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00002000:00000001:5.0:1496184950.857819:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.857820:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.857823:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.857827:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.857830:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.857831:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.857832:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.857832:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.857833:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.857834:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.857837:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57715 00000100:00000040:5.0:1496184950.857839:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.857840:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671280720 : -131872038270896 : ffff88102600c850) 00000100:00000040:5.0:1496184950.857842:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.857847:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.857848:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.857850:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.857852:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.857853:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.857855:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.857856:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857857:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.857857:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.857859:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.857860:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.857861:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.857862:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.857863:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857864:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.857866:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.857867:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.857868:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184950.857869:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.857871:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857872:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.857873:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.857875:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.857879:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.857882:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.857882:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.857885:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.857887:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.857888:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.857889:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.857903:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.857905:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.857906:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.857908:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.857909:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.857910:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.857910:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.857911:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.857913:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.857914:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.857915:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.857917:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.857938:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.857944:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857946:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.857948:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.857950:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.857951:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.857952:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.857953:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.857954:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.857955:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.857957:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.857961:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.857962:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.857990:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff140-0x592def0cff140 00000100:00000001:5.0:1496184950.857992:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.869911:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.869912:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.880882:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.880885:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.880886:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.891865:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.891924:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.891943:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.891945:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.891949:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.891950:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.891952:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.891954:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.891987:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.891989:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.891991:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.891992:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.891993:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.891994:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.891995:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.891996:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.891997:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.891998:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.891999:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.892001:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.892002:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.892004:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.892004:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.892006:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68ec00. 02000000:00000001:5.0:1496184950.892007:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.892009:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.892010:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.892013:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.892018:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.892019:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.892020:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.892021:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.892022:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.892024:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.892030:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 34181us (34333us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.892037:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57715 00000100:00000040:5.0:1496184950.892039:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.892040:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.892041:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.892044:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600c850 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.892049:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.892050:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600c850 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.892052:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.892052:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.892053:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.892055:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.892056:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.892057:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.892058:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.892058:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600c850. 00002000:00000001:5.0:1496184950.892060:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.892061:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.892062:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb680. 00000020:00000010:5.0:1496184950.892064:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.892066:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.892067:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.892069:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.892071:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.892076:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.892077:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.902989:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.902991:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.902994:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c7850 x1568861104107856 msgsize 520 00000100:00100000:6.0:1496184950.902999:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.903000:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.903002:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.903034:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.903037:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.903038:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.903039:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.903040:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.903041:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.903044:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.903046:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107856 00000020:00000001:5.0:1496184950.903047:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.903048:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.903049:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.903050:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.903052:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.903054:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.903056:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.903056:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.903058:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.903060:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb580. 00000020:00000010:5.0:1496184950.903061:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.903063:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.903064:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.903065:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.903067:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.903070:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c7850 x1568861104107856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.903075:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.903076:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903081:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.903083:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.903084:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.903084:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.903085:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.903086:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.903087:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.903089:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43440 00000100:00000040:5.0:1496184950.903090:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.903092:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676240976 : -131872033310640 : ffff8810264c7850) 00000100:00000040:5.0:1496184950.903094:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c7850 x1568861104107856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.903099:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.903100:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.903102:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107856:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.903104:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107856 00000020:00000001:5.0:1496184950.903105:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.903107:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.903110:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.903113:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.903114:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.903115:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68ec00. 02000000:00000001:5.0:1496184950.903116:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903118:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.903120:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.903122:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.903124:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.903125:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.903127:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.903129:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.903132:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.903133:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.903134:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.903135:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.903137:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.903138:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.903139:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.903140:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.903142:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.903143:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.903144:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.903145:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.903146:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.903150:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.903150:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.903152:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.903154:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.903154:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.903157:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.903158:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.903159:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.903160:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.903162:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.903163:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.903165:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.903166:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.903167:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.903169:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.903170:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.903171:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.903172:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.903174:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.903175:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.903176:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.903177:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.903179:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.903180:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.903181:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903183:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.903185:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.903188:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.903189:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.903191:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.903192:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.903193:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903194:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.903196:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.903197:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.903199:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.903200:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.903201:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903203:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.903204:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.903205:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.903206:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.903207:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.903208:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.903209:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.903211:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.903213:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.903214:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.903215:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903217:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.903218:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.903219:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.903220:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.903221:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.903222:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.903223:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903224:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.903225:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.903226:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107856 00010000:00000001:5.0:1496184950.903228:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.903230:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c7850 x1568861104107856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.903235:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.903236:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.903237:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.903239:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.903241:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.903243:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.903244:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.903245:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.903246:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903247:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.903249:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.903251:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107856, offset 0 00000100:00000001:5.0:1496184950.903266:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903267:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.903268:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.903269:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.903270:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.903271:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903272:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.903274:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c7850 x1568861104107856/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.903279:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107856:12345-10.37.248.124@o2ib1:8 Request procesed in 179us (284us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.903284:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43440 00000100:00000040:5.0:1496184950.903285:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.903286:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.903287:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.903289:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb580. 00000020:00000010:5.0:1496184950.903290:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.903292:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.903293:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.903295:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.903297:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.903300:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.903301:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.903345:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.903347:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.903348:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68ec00. 02000000:00000001:7.0:1496184950.903351:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.903352:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.903850:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.903852:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.903854:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600cc50 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.903860:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.903861:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.903863:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.903905:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.903908:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.903909:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.903911:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.903912:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.903913:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.903915:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.903917:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.903919:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.903920:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.903921:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.903922:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.903923:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.903925:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.903927:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.903928:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.903930:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.903931:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb780. 00000020:00000010:5.0:1496184950.903933:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.903935:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.903936:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.903937:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.903938:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.903939:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.903940:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.903942:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.903943:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.903944:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.903945:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.903946:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.903947:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.903948:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.903949:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.903949:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.903950:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903951:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.903952:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903953:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.903954:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.903956:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.903957:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.903958:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.903961:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.903967:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.903968:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.903970:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.903970:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.903971:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.903973:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.903974:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.903975:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.903975:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.903976:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00002000:00000001:5.0:1496184950.903977:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903978:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903981:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.903985:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.903987:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.903988:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.903989:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.903990:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.903991:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.903991:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.903994:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57716 00000100:00000040:5.0:1496184950.903995:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.903997:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671281744 : -131872038269872 : ffff88102600cc50) 00000100:00000040:5.0:1496184950.903999:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.904004:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.904005:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.904007:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.904009:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.904010:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.904012:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.904012:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.904014:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.904014:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.904016:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.904017:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.904018:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.904018:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.904019:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.904020:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.904022:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.904023:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.904024:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e689000. 02000000:00000001:5.0:1496184950.904025:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.904027:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.904028:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.904029:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.904031:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.904035:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.904038:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.904038:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.904041:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.904042:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.904043:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.904044:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.904046:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.904047:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.904048:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.904050:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.904052:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.904052:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.904053:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.904054:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.904056:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.904056:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.904057:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.904060:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.904081:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.904087:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.904089:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.904091:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.904093:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.904094:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.904095:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.904096:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.904097:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.904098:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.904100:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.904104:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.904105:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.904136:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff160-0x592def0cff160 00000100:00000001:5.0:1496184950.904138:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.905969:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.905971:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.905977:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.905981:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.905982:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.905987:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.906026:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.906033:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.906035:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.906039:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.906040:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.906042:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.906044:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.906077:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.906079:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.906081:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.906082:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.906082:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.906084:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.906085:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.906086:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.906087:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.906088:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.906089:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.906090:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.906092:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.906093:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.906094:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.906096:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e689000. 02000000:00000001:5.0:1496184950.906097:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.906099:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.906100:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.906103:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.906108:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.906109:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.906110:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.906110:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.906112:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.906114:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.906119:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 2114us (2263us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.906124:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57716 00000100:00000040:5.0:1496184950.906126:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.906128:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.906128:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.906131:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600cc50 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.906136:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.906137:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600cc50 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.906139:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.906140:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.906141:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.906142:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.906144:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.906144:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.906145:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.906146:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600cc50. 00002000:00000001:5.0:1496184950.906147:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.906148:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.906149:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb780. 00000020:00000010:5.0:1496184950.906151:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.906153:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.906154:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.906155:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.906158:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.906163:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.906164:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.934617:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.934619:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.934621:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6850 x1568861104107888 msgsize 520 00000100:00100000:7.0:1496184950.934626:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.934632:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.934634:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.934666:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.934669:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.934670:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.934671:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.934672:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.934673:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.934676:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.934678:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107888 00000020:00000001:5.0:1496184950.934679:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.934680:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.934681:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.934682:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.934684:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.934685:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.934688:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.934688:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.934690:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.934692:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eba80. 00000020:00000010:5.0:1496184950.934693:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.934695:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.934696:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.934697:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.934699:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.934702:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6850 x1568861104107888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.934707:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.934708:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934712:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.934714:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.934715:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.934716:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.934717:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.934717:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.934718:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.934721:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43441 00000100:00000040:5.0:1496184950.934722:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.934723:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676236880 : -131872033314736 : ffff8810264c6850) 00000100:00000040:5.0:1496184950.934725:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6850 x1568861104107888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.934731:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.934731:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.934733:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107888:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.934735:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107888 00000020:00000001:5.0:1496184950.934736:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.934738:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.934741:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.934744:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.934745:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.934746:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e689000. 02000000:00000001:5.0:1496184950.934747:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934748:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.934751:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.934753:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.934754:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.934756:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.934757:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.934759:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.934762:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.934763:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.934764:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.934765:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.934766:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.934767:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.934768:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.934769:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.934771:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.934772:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.934773:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.934774:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.934775:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.934778:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.934779:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.934780:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.934782:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.934783:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.934785:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.934786:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.934787:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.934788:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.934790:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.934791:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.934793:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.934794:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.934794:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.934796:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.934797:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.934798:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.934799:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.934801:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.934802:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.934803:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.934804:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.934805:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.934807:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.934808:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934809:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.934812:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.934814:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.934815:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.934817:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.934818:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.934819:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934820:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.934822:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.934823:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.934824:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.934825:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.934826:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934828:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.934828:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.934829:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.934830:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.934832:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.934832:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.934834:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.934835:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.934836:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.934838:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.934839:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.934840:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.934841:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.934843:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.934844:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.934845:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.934845:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.934846:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.934847:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.934849:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.934850:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107888 00010000:00000001:5.0:1496184950.934851:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.934853:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6850 x1568861104107888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.934859:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.934859:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.934860:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.934863:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.934864:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.934866:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.934867:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.934868:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.934869:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934870:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.934872:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.934874:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107888, offset 0 00000100:00000001:5.0:1496184950.934884:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.934885:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.934887:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.934888:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.934889:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.934901:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.934902:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.934904:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6850 x1568861104107888/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.934909:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107888:12345-10.37.248.124@o2ib1:8 Request procesed in 177us (286us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.934914:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43441 00000100:00000040:5.0:1496184950.934915:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.934917:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.934917:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.934919:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eba80. 00000020:00000010:5.0:1496184950.934921:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.934922:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.934924:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.934925:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.934928:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.934931:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.934932:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.934980:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.934982:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.934984:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e689000. 02000000:00000001:6.0:1496184950.934987:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.934987:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.935438:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.935440:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.935442:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600d450 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.935447:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.935448:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.935450:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.935480:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.935482:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.935484:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.935485:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.935486:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.935487:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.935490:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.935491:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.935493:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.935494:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.935495:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.935496:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.935498:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.935499:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.935502:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.935502:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.935504:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.935506:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6ebc80. 00000020:00000010:5.0:1496184950.935508:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.935510:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.935511:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.935512:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.935513:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.935514:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.935515:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.935517:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.935518:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.935519:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.935520:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.935521:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.935522:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.935523:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.935524:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.935525:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.935525:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935526:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.935528:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935529:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935530:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.935531:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.935532:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.935533:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.935536:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.935542:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.935543:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.935545:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.935545:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.935546:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.935548:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.935549:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.935550:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.935551:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.935551:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00002000:00000001:5.0:1496184950.935553:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.935554:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.935556:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.935560:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.935562:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.935563:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.935564:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.935565:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.935566:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.935566:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.935569:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57717 00000100:00000040:5.0:1496184950.935570:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.935572:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671283792 : -131872038267824 : ffff88102600d450) 00000100:00000040:5.0:1496184950.935574:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.935579:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.935580:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.935582:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.935584:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.935585:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.935586:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.935587:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935588:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.935589:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.935590:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.935591:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.935592:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.935593:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.935594:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935595:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.935597:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.935598:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.935599:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e68c000. 02000000:00000001:5.0:1496184950.935600:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.935601:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935603:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.935603:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.935605:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.935610:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.935612:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.935613:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.935615:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.935617:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.935618:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.935619:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.935620:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.935622:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.935623:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.935625:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.935626:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.935627:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.935627:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.935628:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.935630:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.935631:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.935632:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.935634:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.935655:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.935661:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935663:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.935665:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.935667:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.935668:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.935669:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.935670:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.935671:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb7400. 00000100:00000010:5.0:1496184950.935673:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.935674:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.935679:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.935680:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.935706:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff180-0x592def0cff180 00000100:00000001:5.0:1496184950.935708:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.937329:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.937331:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.937349:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.937354:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.937354:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb7400 00000100:00000001:3.0:1496184950.937359:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.937402:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.937415:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.937417:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.937421:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.937422:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.937424:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.937426:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.937459:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.937461:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.937463:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.937464:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.937465:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.937466:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.937467:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.937469:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.937469:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.937470:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.937471:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.937473:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.937474:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb7400. 00000100:00000001:5.0:1496184950.937476:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.937477:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.937478:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e68c000. 02000000:00000001:5.0:1496184950.937479:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.937481:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.937483:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.937485:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.937490:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.937492:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.937493:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.937493:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.937494:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.937496:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.937502:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1922us (2058us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.937507:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57717 00000100:00000040:5.0:1496184950.937509:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.937510:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.937511:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.937514:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600d450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.937519:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.937520:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600d450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.937521:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.937522:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.937523:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.937525:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.937526:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.937527:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.937528:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.937528:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600d450. 00002000:00000001:5.0:1496184950.937530:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.937531:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.937532:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6ebc80. 00000020:00000010:5.0:1496184950.937534:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.937535:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.937537:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.937538:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.937541:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.937546:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.937547:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:2.0:1496184950.957632:0:6131:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:2.0:1496184950.957634:0:6131:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_create 00000100:00000040:2.0:1496184950.957637:0:6131:0:(events.c:347:request_in_callback()) incoming req@ffff8800734f5450 x1568861152397744 msgsize 224 00000100:00100000:2.0:1496184950.957643:0:6131:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.197@o2ib1 (source: 12345-10.37.248.197@o2ib1) 00000100:00000040:2.0:1496184950.957645:0:6131:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:2.0:1496184950.957648:0:6131:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:1.0:1496184950.957682:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184950.957685:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:1.0:1496184950.957689:0:14306:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:1.0:1496184950.957691:0:14306:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:1.0:1496184950.957693:0:14306:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:1.0:1496184950.957695:0:14306:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1496184950.957700:0:14306:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:1.0:1496184950.957703:0:14306:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861152397744 00000020:00000001:1.0:1496184950.957709:0:14306:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:1.0:1496184950.957710:0:14306:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf611 00000020:00000001:1.0:1496184950.957712:0:14306:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:1.0:1496184950.957714:0:14306:0:(genops.c:818:class_export_get()) GETting export ffff880174bcf000 : new refcount 5 00000020:00000001:1.0:1496184950.957716:0:14306:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612138567725056 : -131935141826560 : ffff880174bcf000) 00000020:00000001:1.0:1496184950.957718:0:14306:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612138567725056 : -131935141826560 : ffff880174bcf000) 00000100:00000001:1.0:1496184950.957722:0:14306:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:1.0:1496184950.957724:0:14306:0:(service.c:1128:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:1.0:1496184950.957726:0:14306:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff88100fc67a00. 00000020:00000010:1.0:1496184950.957730:0:14306:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100eeb1900. 00000020:00000010:1.0:1496184950.957733:0:14306:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88100eeb79c0. 00000100:00000040:1.0:1496184950.957736:0:14306:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_create at +1s 00000100:00000001:1.0:1496184950.957737:0:14306:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:1.0:1496184950.957739:0:14306:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:1.0:1496184950.957740:0:14306:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957744:0:14306:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957750:0:14306:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:2.0:1496184950.957754:0:13733:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184950.957754:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184950.957755:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:2.0:1496184950.957756:0:13733:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00004000:00000001:1.0:1496184950.957756:0:14306:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:1.0:1496184950.957756:0:14306:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:2.0:1496184950.957758:0:13733:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:1.0:1496184950.957758:0:14306:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:2.0:1496184950.957759:0:13733:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00000001:1.0:1496184950.957759:0:14306:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:1.0:1496184950.957763:0:14306:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.197@o2ib1, seq: 960 00000100:00000001:2.0:1496184950.957765:0:13733:0:(service.c:1841:ptlrpc_server_request_get()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:1.0:1496184950.957765:0:14306:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff880174bcf000 : new rpc_count 1 00000100:00000001:1.0:1496184950.957766:0:14306:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612134248797264 : -131939460754352 : ffff8800734f5450) 00000100:00000001:2.0:1496184950.957767:0:13733:0:(service.c:2043:ptlrpc_server_handle_request()) Process leaving (rc=0 : 0 : 0) 00004000:00000001:2.0:1496184950.957769:0:13733:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:2.0:1496184950.957770:0:13733:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000040:1.0:1496184950.957770:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8800734f5450 x1568861152397744/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:701/0 lens 224/0 e 0 to 0 dl 1496184956 ref 1 fl New:/0/ffffffff rc 0/-1 00000100:00000001:1.0:1496184950.957777:0:14306:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:1.0:1496184950.957778:0:14306:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:1.0:1496184950.957780:0:14306:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397744:12345-10.37.248.197@o2ib1:13 00000100:00000200:1.0:1496184950.957782:0:14306:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861152397744 00000020:00000001:1.0:1496184950.957784:0:14306:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:1.0:1496184950.957789:0:14306:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:1.0:1496184950.957790:0:14306:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184950.957792:0:14306:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:1.0:1496184950.957793:0:14306:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284592 : -1589267024 : ffffffffa145b5b0) 00000020:00000001:1.0:1496184950.957795:0:14306:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:1.0:1496184950.957797:0:14306:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:1.0:1496184950.957798:0:14306:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:1.0:1496184950.957799:0:14306:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:1.0:1496184950.957801:0:14306:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957804:0:14306:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:1.0:1496184950.957806:0:14306:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:1.0:1496184950.957808:0:14306:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 624 at ffff880174bcf800. 02000000:00000001:1.0:1496184950.957810:0:14306:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957811:0:14306:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:1.0:1496184950.957813:0:14306:0:(ofd_dev.c:1916:ofd_statfs_hdl()) Process entered 00002000:00000001:1.0:1496184950.957816:0:14306:0:(ofd_obd.c:725:ofd_statfs()) Process entered 00002000:00000001:1.0:1496184950.957818:0:14306:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:1.0:1496184950.957825:0:14306:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000024:1.0:1496184950.957826:0:14306:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 270144 pending 0 free 80207056896 avail 76094279680 00002000:00000020:1.0:1496184950.957829:0:14306:0:(tgt_grant.c:150:tgt_check_export_grants()) lustre-OST0000: cli lustre-MDT0000-mdtlov_UUID/ffff880174bcf000 dirty 0 pend 0 grant 0 00002000:00000020:1.0:1496184950.957832:0:14306:0:(tgt_grant.c:144:tgt_check_export_grants()) lustre-OST0000: processing self export: 270144 0 0 00002000:00000020:1.0:1496184950.957833:0:14306:0:(tgt_grant.c:150:tgt_check_export_grants()) lustre-OST0000: cli lustre-OST0000_UUID/ffff8807fc3c4c00 dirty 0 pend 0 grant 270144 00002000:00000020:1.0:1496184950.957835:0:14306:0:(ofd_obd.c:760:ofd_statfs()) 19595366 blocks: 19581801 free, 18577640 avail; 1173120 objects: 1172610 free; state 0 00002000:00000001:1.0:1496184950.957837:0:14306:0:(ofd_obd.c:794:ofd_statfs()) Process leaving 00002000:00000001:1.0:1496184950.957839:0:14306:0:(ofd_dev.c:1932:ofd_statfs_hdl()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:1.0:1496184950.957841:0:14306:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294967372, transno 0, xid 1568861152397744 00010000:00000001:1.0:1496184950.957843:0:14306:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:1.0:1496184950.957846:0:14306:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8800734f5450 x1568861152397744/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:701/0 lens 224/368 e 0 to 0 dl 1496184956 ref 1 fl Interpret:/0/0 rc 0/0 00010000:00000001:1.0:1496184950.957852:0:14306:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:1.0:1496184950.957854:0:14306:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:1.0:1496184950.957856:0:14306:0:(import.c:1678:at_measured()) add 1 to ffff8807f6ee7d50 time=104 v=1 (1 1 1 1) 00000100:00000001:1.0:1496184950.957859:0:14306:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:1.0:1496184950.957861:0:14306:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff88103ef15780 refcount 6 to 10.37.248.197@o2ib1 00000100:00000001:1.0:1496184950.957862:0:14306:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612202089699200 : -131871619852416 : ffff88103ef15780) 02000000:00000001:1.0:1496184950.957865:0:14306:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:1.0:1496184950.957866:0:14306:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957868:0:14306:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:1.0:1496184950.957870:0:14306:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.197@o2ib1 00000100:00000200:1.0:1496184950.957874:0:14306:0:(niobuf.c:85:ptl_send_buf()) Sending 336 bytes to portal 4, xid 1568861152397744, offset 192 00000100:00000001:1.0:1496184950.957903:0:14306:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1496184950.957904:0:14306:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:1.0:1496184950.957906:0:14306:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff88103ef15780 refcount 5 to 10.37.248.197@o2ib1 00000100:00000001:1.0:1496184950.957907:0:14306:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:1.0:1496184950.957908:0:14306:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:1.0:1496184950.957909:0:14306:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:1.0:1496184950.957910:0:14306:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:1.0:1496184950.957912:0:14306:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8800734f5450 x1568861152397744/t0(0) o13->lustre-MDT0000-mdtlov_UUID@10.37.248.197@o2ib1:701/0 lens 224/336 e 0 to 0 dl 1496184956 ref 1 fl Interpret:/0/0 rc 0/0 00000100:00000001:2.0:1496184950.957914:0:6132:0:(events.c:394:reply_out_callback()) Process entered 00000100:00000001:2.0:1496184950.957914:0:6132:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00100000:1.0:1496184950.957918:0:14306:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_create00:lustre-MDT0000-mdtlov_UUID+5:13022:x1568861152397744:12345-10.37.248.197@o2ib1:13 Request procesed in 140us (280us total) trans 0 rc 0/0 00000100:00100000:1.0:1496184950.957923:0:14306:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.197@o2ib1, seq: 960 00000100:00000040:1.0:1496184950.957925:0:14306:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880174bcf000 : new rpc_count 0 00000100:00000001:1.0:1496184950.957926:0:14306:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:1.0:1496184950.957927:0:14306:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:1.0:1496184950.957930:0:14306:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100eeb1900. 00000020:00000010:1.0:1496184950.957932:0:14306:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88100eeb79c0. 00000020:00000010:1.0:1496184950.957933:0:14306:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff88100fc67a00. 00000020:00000040:1.0:1496184950.957936:0:14306:0:(genops.c:828:class_export_put()) PUTting export ffff880174bcf000 : new refcount 4 02000000:00000001:1.0:1496184950.957938:0:14306:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:1.0:1496184950.957940:0:14306:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 624 at ffff880174bcf800. 02000000:00000001:1.0:1496184950.957941:0:14306:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:1.0:1496184950.957942:0:14306:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:1.0:1496184950.957945:0:14306:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 7 00004000:00000001:1.0:1496184950.957950:0:14306:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:1.0:1496184950.957951:0:14306:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.965591:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.965592:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:6.0:1496184950.965595:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c6050 x1568861104107920 msgsize 520 00000100:00100000:6.0:1496184950.965600:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.965601:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.965604:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.965634:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.965636:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.965638:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.965639:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.965641:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.965642:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.965644:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.965646:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107920 00000020:00000001:5.0:1496184950.965648:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.965649:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.965650:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.965651:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.965653:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.965655:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.965657:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.965658:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.965659:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.965661:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb380. 00000020:00000010:5.0:1496184950.965663:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.965665:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.965666:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.965667:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.965669:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.965672:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c6050 x1568861104107920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.965681:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.965682:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965687:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.965689:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.965690:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.965690:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.965691:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.965692:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.965693:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.965695:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43442 00000100:00000040:5.0:1496184950.965697:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.965698:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676234832 : -131872033316784 : ffff8810264c6050) 00000100:00000040:5.0:1496184950.965701:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c6050 x1568861104107920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.965706:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.965706:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.965708:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107920:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.965711:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107920 00000020:00000001:5.0:1496184950.965712:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.965714:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.965717:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.965720:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.965721:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.965722:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e68c000. 02000000:00000001:5.0:1496184950.965723:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965725:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.965727:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.965729:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.965731:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.965732:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.965733:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.965736:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.965738:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.965739:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.965740:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.965741:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.965742:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.965743:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.965744:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.965745:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.965747:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.965748:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.965749:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.965750:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.965751:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.965754:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.965755:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.965757:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.965758:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.965759:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.965761:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.965763:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.965763:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.965764:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.965766:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.965768:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.965769:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.965770:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.965771:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.965772:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.965774:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.965775:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.965776:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.965777:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.965779:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.965780:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.965781:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.965782:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.965784:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.965785:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965786:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.965789:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.965791:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.965792:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.965794:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.965795:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.965796:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965797:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.965799:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.965800:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.965801:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.965802:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.965804:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965805:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.965806:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.965807:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.965808:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.965809:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.965810:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.965811:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.965813:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.965814:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.965815:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.965816:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.965818:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.965819:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.965821:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.965822:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.965823:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.965824:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.965825:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.965826:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.965827:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.965828:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107920 00010000:00000001:5.0:1496184950.965830:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.965832:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c6050 x1568861104107920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.965837:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.965838:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.965839:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.965841:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.965843:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.965844:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.965845:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.965847:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.965848:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965849:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.965850:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.965852:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107920, offset 0 00000100:00000001:5.0:1496184950.965863:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.965864:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.965866:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.965867:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.965868:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.965868:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.965870:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.965871:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c6050 x1568861104107920/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.965877:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107920:12345-10.37.248.124@o2ib1:8 Request procesed in 170us (281us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.965882:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43442 00000100:00000040:5.0:1496184950.965883:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.965885:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.965885:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.965887:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb380. 00000020:00000010:5.0:1496184950.965889:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.965906:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.965907:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.965909:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.965911:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.965914:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.965915:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:6.0:1496184950.965951:0:6134:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:6.0:1496184950.965954:0:6134:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:6.0:1496184950.965955:0:6134:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e68c000. 02000000:00000001:6.0:1496184950.965958:0:6134:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:6.0:1496184950.965959:0:6134:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:6.0:1496184950.966359:0:6134:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:6.0:1496184950.966361:0:6134:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:6.0:1496184950.966363:0:6134:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e050 x1568861102817424 msgsize 608 00000100:00100000:6.0:1496184950.966368:0:6134:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:6.0:1496184950.966369:0:6134:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:6.0:1496184950.966371:0:6134:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.966402:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.966404:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.966406:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.966407:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.966408:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.966409:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.966411:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.966413:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.966415:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.966416:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.966417:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.966418:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.966420:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.966421:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.966423:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.966424:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.966426:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.966428:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb900. 00000020:00000010:5.0:1496184950.966429:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.966431:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.966433:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.966433:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.966435:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.966436:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.966437:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.966438:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.966439:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.966441:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.966441:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.966443:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.966443:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.966445:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.966445:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.966446:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.966447:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966448:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.966449:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966450:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966451:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.966452:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.966453:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.966454:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.966457:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.966464:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.966464:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.966466:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.966467:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.966468:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.966469:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.966471:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.966472:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.966472:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.966473:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00002000:00000001:5.0:1496184950.966474:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.966475:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.966477:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.966482:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.966484:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.966485:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.966486:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.966486:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.966487:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.966488:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.966490:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57718 00000100:00000040:5.0:1496184950.966492:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.966493:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671286864 : -131872038264752 : ffff88102600e050) 00000100:00000040:5.0:1496184950.966495:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.966501:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.966501:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.966503:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.966505:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.966506:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.966508:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.966509:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966510:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.966510:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.966512:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.966513:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.966514:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.966515:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.966515:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966516:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.966518:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.966519:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.966520:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e688c00. 02000000:00000001:5.0:1496184950.966522:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.966523:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966524:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.966525:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.966527:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.966531:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.966534:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.966535:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.966537:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.966539:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.966540:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.966540:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.966542:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.966544:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.966544:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.966546:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.966548:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.966549:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.966549:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.966550:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.966552:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.966553:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.966554:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.966556:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.966577:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.966582:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966584:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.966586:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.966588:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.966589:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.966590:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.966591:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.966592:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.966594:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.966595:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.966600:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.966601:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.966627:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff1a0-0x592def0cff1a0 00000100:00000001:5.0:1496184950.966629:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.967966:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.967969:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.967989:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184950.967993:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.967993:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184950.967998:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00010000:00020000:5.0:1496184950.968036:0:17513:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ network error on bulk WRITE req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00010000:00000001:5.0:1496184950.968042:0:17513:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.968044:0:17513:0:(obd_class.h:1141:obd_commitrw()) Process entered 00002000:00000001:5.0:1496184950.968048:0:17513:0:(ofd_io.c:1023:ofd_commitrw_write()) Process entered 00002000:00000001:5.0:1496184950.968049:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.968051:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.968053:0:17513:0:(ofd_io.c:1035:ofd_commitrw_write()) Process leaving via out (rc=18446744073709551506 : -110 : 0xffffffffffffff92) 00000001:00000001:5.0:1496184950.968086:0:17513:0:(osd_io.c:486:osd_bufs_put()) Process leaving (rc=0 : 0 : 0) 00000020:00000002:5.0:1496184950.968088:0:17513:0:(lu_object.c:158:lu_object_put()) Add ffff88080dbd9aa8 to site lru. hash: ffff88105cc48600, bkt: ffffc90037d1b018, lru_len: 1 00002000:00000001:5.0:1496184950.968090:0:17513:0:(tgt_grant.c:1373:tgt_grant_commit()) Process entered 00002000:00000001:5.0:1496184950.968091:0:17513:0:(tgt_grant.c:1427:tgt_grant_commit()) Process leaving 00002000:00000001:5.0:1496184950.968092:0:17513:0:(ofd_io.c:1164:ofd_commitrw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00002000:00000001:5.0:1496184950.968093:0:17513:0:(ofd_io.c:1289:ofd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.968094:0:17513:0:(obd_class.h:1148:obd_commitrw()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:5.0:1496184950.968096:0:17513:0:(tgt_handler.c:1646:tgt_brw_unlock()) Process entered 00000020:00000001:5.0:1496184950.968096:0:17513:0:(tgt_handler.c:1655:tgt_brw_unlock()) Process leaving 00000100:00000001:5.0:1496184950.968097:0:17513:0:(client.c:257:ptlrpc_free_bulk()) Process entered 00000020:00000040:5.0:1496184950.968098:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000100:00000010:5.0:1496184950.968100:0:17513:0:(client.c:278:ptlrpc_free_bulk()) kfreed '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000100:00000010:5.0:1496184950.968101:0:17513:0:(client.c:282:ptlrpc_free_bulk()) kfreed 'desc': 280 at ffff881010fb6e00. 00000100:00000001:5.0:1496184950.968103:0:17513:0:(client.c:283:ptlrpc_free_bulk()) Process leaving 02000000:00000001:5.0:1496184950.968104:0:17513:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:5.0:1496184950.968105:0:17513:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 704 at ffff88100e688c00. 02000000:00000001:5.0:1496184950.968106:0:17513:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000020:02000400:5.0:1496184950.968108:0:17513:0:(tgt_handler.c:2347:tgt_brw_write()) lustre-OST0001: Bulk IO write error with 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1), client will retry: rc = -110 00000020:00000001:5.0:1496184950.968110:0:17513:0:(tgt_handler.c:2350:tgt_brw_write()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00010000:00000080:5.0:1496184950.968112:0:17513:0:(ldlm_lib.c:2881:target_committed_to_req()) @@@ not sending last_committed update (0/1) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00010000:00000040:5.0:1496184950.968117:0:17513:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861102817424 00010000:00000001:5.0:1496184950.968119:0:17513:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000001:5.0:1496184950.968120:0:17513:0:(ldlm_lib.c:2951:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.968120:0:17513:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.968122:0:17513:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.968123:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/ffffffff rc -110/-1 00000100:00100000:5.0:1496184950.968129:0:17513:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 Request procesed in 1627us (1764us total) trans 0 rc -110/-999 00000100:00100000:5.0:1496184950.968134:0:17513:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 57718 00000100:00000040:5.0:1496184950.968136:0:17513:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.968137:0:17513:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00002000:00000001:5.0:1496184950.968138:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.968141:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e050 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Complete:/2/ffffffff rc -110/-1 00002000:00000001:5.0:1496184950.968146:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.968147:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e050 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.968148:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.968149:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.968150:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.968152:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.968153:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.968154:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.968155:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.968155:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e050. 00002000:00000001:5.0:1496184950.968156:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.968158:0:17513:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.968159:0:17513:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb900. 00000020:00000010:5.0:1496184950.968161:0:17513:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.968162:0:17513:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.968164:0:17513:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.968165:0:17513:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.968167:0:17513:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 6 00004000:00000001:5.0:1496184950.968173:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.968174:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.996193:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.996195:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost 00000100:00000040:7.0:1496184950.996197:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff8810264c5c50 x1568861104107952 msgsize 520 00000100:00100000:7.0:1496184950.996202:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.996203:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.996205:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.996240:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.996243:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.996245:0:13731:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.996246:0:13731:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.996247:0:13731:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.996248:0:13731:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.996250:0:13731:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.996252:0:13731:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861104107952 00000020:00000001:5.0:1496184950.996254:0:13731:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.996254:0:13731:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.996256:0:13731:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.996257:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.996258:0:13731:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.996260:0:13731:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.996262:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.996263:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.996264:0:13731:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.996266:0:13731:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb280. 00000020:00000010:5.0:1496184950.996268:0:13731:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000001:5.0:1496184950.996269:0:13731:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.996270:0:13731:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000100:00000001:5.0:1496184950.996271:0:13731:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:5.0:1496184950.996273:0:13731:0:(nrs.c:864:ptlrpc_nrs_hpreq_add_nolock()) Process entered 00000100:00000200:5.0:1496184950.996276:0:13731:0:(nrs.c:870:ptlrpc_nrs_hpreq_add_nolock()) @@@ high priority req req@ffff8810264c5c50 x1568861104107952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.996281:0:13731:0:(nrs.c:872:ptlrpc_nrs_hpreq_add_nolock()) Process leaving 00000100:00000001:5.0:1496184950.996282:0:13731:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996286:0:13731:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.996288:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.996289:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.996290:0:13731:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.996291:0:13731:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.996292:0:13731:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.996292:0:13731:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.996295:0:13731:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 43443 00000100:00000040:5.0:1496184950.996296:0:13731:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.996298:0:13731:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201676233808 : -131872033317808 : ffff8810264c5c50) 00000100:00000040:5.0:1496184950.996300:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff8810264c5c50 x1568861104107952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/0 e 0 to 0 dl 1496184970 ref 1 fl New:H/0/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.996305:0:13731:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.996306:0:13731:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.996307:0:13731:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107952:12345-10.37.248.124@o2ib1:8 00000100:00000200:5.0:1496184950.996309:0:13731:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861104107952 00000020:00000001:5.0:1496184950.996311:0:13731:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00010000:00000001:5.0:1496184950.996313:0:13731:0:(ldlm_lib.c:966:target_handle_connect()) Process entered 00000020:00000040:5.0:1496184950.996316:0:13731:0:(obd_config.c:753:class_incref()) incref lustre-OST0001 (ffff8810117a0000) now 8 00000100:00000001:5.0:1496184950.996318:0:13731:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.996319:0:13731:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.996320:0:13731:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 672 at ffff88100e688c00. 02000000:00000001:5.0:1496184950.996322:0:13731:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996323:0:13731:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.996325:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.996327:0:13731:0:(ldlm_lib.c:748:target_handle_reconnect()) Process entered 00010000:02000400:5.0:1496184950.996329:0:13731:0:(ldlm_lib.c:779:target_handle_reconnect()) lustre-OST0001: Client 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) reconnecting 00010000:00000001:5.0:1496184950.996330:0:13731:0:(ldlm_lib.c:780:target_handle_reconnect()) Process leaving via out_already (rc=0 : 0 : 0x0) 00010000:00000001:5.0:1496184950.996332:0:13731:0:(ldlm_lib.c:830:target_handle_reconnect()) Process leaving (rc=114 : 114 : 72) 00010000:00080000:5.0:1496184950.996334:0:13731:0:(ldlm_lib.c:1239:target_handle_connect()) lustre-OST0001: connection from 935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1 t4294968338 exp ffff8810371fac00 cur 1496184950 last 1496184949 00010000:00000001:5.0:1496184950.996336:0:13731:0:(obd_class.h:849:obd_reconnect()) Process entered 00002000:00000001:5.0:1496184950.996337:0:13731:0:(ofd_obd.c:313:ofd_obd_reconnect()) Process entered 00000001:00000001:5.0:1496184950.996338:0:13731:0:(nodemap_handler.c:385:nodemap_add_member()) Process entered 00000001:00000001:5.0:1496184950.996339:0:13731:0:(nodemap_handler.c:264:nodemap_classify_nid()) Process entered 00000001:00000001:5.0:1496184950.996340:0:13731:0:(interval_tree.c:629:interval_search()) Process entered 00000001:00000001:5.0:1496184950.996341:0:13731:0:(interval_tree.c:676:interval_search()) Process leaving (rc=1 : 1 : 1) 00000001:00000040:5.0:1496184950.996342:0:13731:0:(nodemap_handler.c:97:nodemap_getref()) GETting nodemap default(p=ffff881025bfa000) : new refcount 6 00000001:00000001:5.0:1496184950.996343:0:13731:0:(nodemap_handler.c:290:nodemap_classify_nid()) Process leaving (rc=18446612201667010560 : -131872042541056 : ffff881025bfa000) 00000001:00000001:5.0:1496184950.996345:0:13731:0:(nodemap_member.c:99:nm_member_add()) Process entered 00000001:00000001:5.0:1496184950.996346:0:13731:0:(nodemap_member.c:114:nm_member_add()) Process leaving (rc=0 : 0 : 0) 00000001:00000040:5.0:1496184950.996347:0:13731:0:(nodemap_handler.c:113:nodemap_putref()) PUTting nodemap default(p=ffff881025bfa000) : new refcount 5 00000001:00000001:5.0:1496184950.996348:0:13731:0:(nodemap_handler.c:406:nodemap_add_member()) Process leaving (rc=0 : 0 : 0) 00002000:00100000:5.0:1496184950.996349:0:13731:0:(ofd_obd.c:157:ofd_parse_connect_data()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_connect_flags: 0x20445af0e3640478 ocd_version: 2093a00 ocd_grant: 8388608 ocd_index: 0 ocd_group 0 00002000:00000001:5.0:1496184950.996352:0:13731:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.996353:0:13731:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.996354:0:13731:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.996356:0:13731:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.996357:0:13731:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.996359:0:13731:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.996360:0:13731:0:(tgt_grant.c:878:tgt_grant_alloc()) Process entered 00002000:00000001:5.0:1496184950.996361:0:13731:0:(tgt_grant.c:904:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0) 00002000:00000020:5.0:1496184950.996362:0:13731:0:(tgt_grant.c:1028:tgt_grant_connect()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 ocd_grant: 9510912 want: 8388608 left: 76084350976 00002000:00000001:5.0:1496184950.996364:0:13731:0:(tgt_grant.c:1030:tgt_grant_connect()) Process leaving 00002000:00100000:5.0:1496184950.996365:0:13731:0:(ofd_obd.c:264:ofd_parse_connect_data()) lustre-OST0001: cli 10.37.248.124@o2ib1 supports cksum type 7, return 7 00002000:00000001:5.0:1496184950.996367:0:13731:0:(ofd_obd.c:286:ofd_parse_connect_data()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.996368:0:13731:0:(ofd_obd.c:74:ofd_export_stats_init()) Process entered 00000020:00000001:5.0:1496184950.996368:0:13731:0:(lprocfs_status_server.c:391:lprocfs_exp_setup()) Process entered 00000020:00000001:5.0:1496184950.996370:0:13731:0:(lprocfs_status_server.c:408:lprocfs_exp_setup()) Process leaving (rc=18446744073709551502 : -114 : ffffffffffffff8e) 00002000:00000001:5.0:1496184950.996372:0:13731:0:(ofd_obd.c:85:ofd_export_stats_init()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.996373:0:13731:0:(ofd_obd.c:330:ofd_obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.996373:0:13731:0:(obd_class.h:859:obd_reconnect()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:5.0:1496184950.996375:0:13731:0:(service.c:1004:ptlrpc_request_change_export()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000020:00000040:5.0:1496184950.996376:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.996377:0:13731:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00000100:00000040:5.0:1496184950.996378:0:13731:0:(service.c:1010:ptlrpc_request_change_export()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.996379:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.996381:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.996382:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996383:0:13731:0:(connection.c:48:ptlrpc_connection_get()) Process entered 00000100:00000001:5.0:1496184950.996386:0:13731:0:(connection.c:53:ptlrpc_connection_get()) Process leaving via out (rc=18446612201682606624 : -131872026944992 : 0xffff881026ad9a20) 00000100:00000040:5.0:1496184950.996389:0:13731:0:(connection.c:84:ptlrpc_connection_get()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00080000:5.0:1496184950.996390:0:13731:0:(import.c:104:ptlrpc_import_enter_resend()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from FULL to RECOVER 00000100:00000001:5.0:1496184950.996392:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.996393:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 2 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.996394:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996395:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.996397:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.996398:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.996399:0:13731:0:(sec.c:1434:sptlrpc_import_sec_adapt()) Process entered 02000000:00000001:5.0:1496184950.996400:0:13731:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Process leaving via out (rc=0 : 0 : 0x0) 02000000:00000001:5.0:1496184950.996402:0:13731:0:(sec.c:1505:sptlrpc_import_sec_adapt()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996403:0:13731:0:(import.c:1460:ptlrpc_import_recovery_state_machine()) Process entered 00000100:00000001:5.0:1496184950.996404:0:13731:0:(recover.c:178:ptlrpc_resend()) Process entered 00000100:00000001:5.0:1496184950.996404:0:13731:0:(recover.c:206:ptlrpc_resend()) Process leaving (rc=0 : 0 : 0) 00000100:00080000:5.0:1496184950.996405:0:13731:0:(import.c:1536:ptlrpc_import_recovery_state_machine()) ffff88100e915800 Ð3rDˆÿÿðj:ˆÿÿ: changing import state from RECOVER to FULL 00000100:00000001:5.0:1496184950.996407:0:13731:0:(obd_class.h:1168:obd_import_event()) Process entered 00000100:00000001:5.0:1496184950.996408:0:13731:0:(obd_class.h:1178:obd_import_event()) Process leaving 00000100:02000000:5.0:1496184950.996409:0:13731:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) lustre-OST0001: Connection restored to 935a409f-52ff-39aa-2890-65a4411c7c57 (at 10.37.248.124@o2ib1) 00000100:00000001:5.0:1496184950.996410:0:13731:0:(import.c:1551:ptlrpc_import_recovery_state_machine()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:5.0:1496184950.996412:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 8 00000020:00000040:5.0:1496184950.996413:0:13731:0:(obd_config.c:770:class_decref()) Decref lustre-OST0001 (ffff8810117a0000) now 7 00010000:00000001:5.0:1496184950.996414:0:13731:0:(ldlm_lib.c:1452:target_handle_connect()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.996416:0:13731:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.996417:0:13731:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284352 : -1589267264 : ffffffffa145b4c0) 00000020:00000001:5.0:1496184950.996418:0:13731:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.996419:0:13731:0:(tgt_handler.c:532:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.996420:0:13731:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.996421:0:13731:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.996422:0:13731:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.996423:0:13731:0:(tgt_handler.c:935:tgt_connect()) Process entered 00000020:00000001:5.0:1496184950.996424:0:13731:0:(tgt_handler.c:955:tgt_connect()) Process leaving (rc=0 : 0 : 0) 00010000:00000040:5.0:1496184950.996425:0:13731:0:(ldlm_lib.c:2884:target_committed_to_req()) last_committed 4294968338, transno 0, xid 1568861104107952 00010000:00000001:5.0:1496184950.996427:0:13731:0:(ldlm_lib.c:2948:target_send_reply()) Process entered 00010000:00000200:5.0:1496184950.996428:0:13731:0:(ldlm_lib.c:2936:target_send_reply_msg()) @@@ sending reply req@ffff8810264c5c50 x1568861104107952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/416 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00010000:00000001:5.0:1496184950.996434:0:13731:0:(ldlm_lib.c:2895:target_pack_pool_reply()) Process entered 00010000:00000001:5.0:1496184950.996435:0:13731:0:(ldlm_lib.c:2914:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:5.0:1496184950.996436:0:13731:0:(import.c:1678:at_measured()) add 1 to ffff881026bdfd50 time=95 v=1 (1 1 1 1) 00000100:00001000:5.0:1496184950.996438:0:13731:0:(niobuf.c:547:ptlrpc_at_set_reply()) No early reply support: flags=0x1 req_flags=0x0 magic=bd00bd3/bd00bd3 len=384 00000100:00000001:5.0:1496184950.996440:0:13731:0:(connection.c:127:ptlrpc_connection_addref()) Process entered 00000100:00000040:5.0:1496184950.996441:0:13731:0:(connection.c:132:ptlrpc_connection_addref()) conn=ffff881026ad9a20 refcount 4 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.996442:0:13731:0:(connection.c:134:ptlrpc_connection_addref()) Process leaving (rc=18446612201682606624 : -131872026944992 : ffff881026ad9a20) 02000000:00000001:5.0:1496184950.996444:0:13731:0:(sec.c:2161:sptlrpc_svc_wrap_reply()) Process entered 02000000:00000001:5.0:1496184950.996444:0:13731:0:(sec.c:2172:sptlrpc_svc_wrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996445:0:13731:0:(niobuf.c:54:ptl_send_buf()) Process entered 00000100:00000040:5.0:1496184950.996447:0:13731:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.37.248.124@o2ib1 00000100:00000200:5.0:1496184950.996449:0:13731:0:(niobuf.c:85:ptl_send_buf()) Sending 384 bytes to portal 4, xid 1568861104107952, offset 0 00000100:00000001:5.0:1496184950.996460:0:13731:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.996461:0:13731:0:(connection.c:91:ptlrpc_connection_put()) Process entered 00000100:00000040:5.0:1496184950.996462:0:13731:0:(connection.c:119:ptlrpc_connection_put()) PUT conn=ffff881026ad9a20 refcount 3 to 10.37.248.124@o2ib1 00000100:00000001:5.0:1496184950.996463:0:13731:0:(connection.c:121:ptlrpc_connection_put()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:5.0:1496184950.996464:0:13731:0:(ldlm_lib.c:2960:target_send_reply()) Process leaving 00000020:00000001:5.0:1496184950.996465:0:13731:0:(tgt_handler.c:475:tgt_handle_request0()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.996466:0:13731:0:(tgt_handler.c:780:tgt_request_handle()) Process leaving 00000100:00000040:5.0:1496184950.996468:0:13731:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "Interpret" -> "Complete" req@ffff8810264c5c50 x1568861104107952/t0(0) o8->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:0/0 lens 520/384 e 0 to 0 dl 1496184970 ref 1 fl Interpret:H/0/0 rc 0/0 00000100:00100000:5.0:1496184950.996473:0:13731:0:(service.c:2146:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost01_001:935a409f-52ff-39aa-2890-65a4411c7c57+8:12082:x1568861104107952:12345-10.37.248.124@o2ib1:8 Request procesed in 167us (274us total) trans 0 rc 0/0 00000100:00100000:5.0:1496184950.996478:0:13731:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.37.248.124@o2ib1, seq: 43443 00000100:00000040:5.0:1496184950.996479:0:13731:0:(service.c:1045:ptlrpc_server_finish_active_request()) RPC PUTting export ffff8810371fac00 : new rpc_count 0 00000100:00000001:5.0:1496184950.996480:0:13731:0:(service.c:1624:ptlrpc_server_hpreq_fini()) Process entered 00000100:00000001:5.0:1496184950.996481:0:13731:0:(service.c:1635:ptlrpc_server_hpreq_fini()) Process leaving 00000020:00000010:5.0:1496184950.996483:0:13731:0:(lu_ucred.c:49:lu_ucred_key_fini()) kfreed 'info': 104 at ffff88100b6eb280. 00000020:00000010:5.0:1496184950.996484:0:13731:0:(tgt_main.c:382:tgt_ses_key_fini()) kfreed 'info': 160 at ffff88104ffc6000. 00000020:00000010:5.0:1496184950.996486:0:13731:0:(lu_object.c:1642:keys_fini()) kfreed 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000040:5.0:1496184950.996487:0:13731:0:(genops.c:828:class_export_put()) PUTting export ffff8810371fac00 : new refcount 7 00000100:00000001:5.0:1496184950.996489:0:13731:0:(service.c:2166:ptlrpc_server_handle_request()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:5.0:1496184950.996490:0:13731:0:(niobuf.c:950:ptlrpc_register_rqbd()) LNetMEAttach: portal 28 00004000:00000001:5.0:1496184950.996494:0:13731:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.996494:0:13731:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00000100:00000001:7.0:1496184950.996549:0:6133:0:(events.c:394:reply_out_callback()) Process entered 02000000:00000001:7.0:1496184950.996551:0:6133:0:(sec.c:2182:sptlrpc_svc_free_rs()) Process entered 02000000:00000010:7.0:1496184950.996553:0:6133:0:(sec_null.c:343:null_free_rs()) kfreed 'rs': 672 at ffff88100e688c00. 02000000:00000001:7.0:1496184950.996556:0:6133:0:(sec.c:2195:sptlrpc_svc_free_rs()) Process leaving 00000100:00000001:7.0:1496184950.996556:0:6133:0:(events.c:405:reply_out_callback()) Process leaving 00000100:00000001:7.0:1496184950.997005:0:6133:0:(events.c:294:request_in_callback()) Process entered 00000100:00000200:7.0:1496184950.997006:0:6133:0:(events.c:304:request_in_callback()) event type 2, status 0, service ost_io 00000100:00000040:7.0:1496184950.997009:0:6133:0:(events.c:347:request_in_callback()) incoming req@ffff88102600e450 x1568861102817424 msgsize 608 00000100:00100000:7.0:1496184950.997014:0:6133:0:(events.c:350:request_in_callback()) peer: 12345-10.37.248.124@o2ib1 (source: 12345-10.37.248.124@o2ib1) 00000100:00000040:7.0:1496184950.997015:0:6133:0:(events.c:359:request_in_callback()) Buffer complete: 63 buffers still posted 00000100:00000001:7.0:1496184950.997017:0:6133:0:(events.c:383:request_in_callback()) Process leaving 00004000:00000001:5.0:1496184950.997049:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.997051:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.997053:0:17513:0:(service.c:1870:ptlrpc_server_handle_req_in()) Process entered 02000000:00000001:5.0:1496184950.997054:0:17513:0:(sec.c:2052:sptlrpc_svc_unwrap_request()) Process entered 00000100:00000001:5.0:1496184950.997055:0:17513:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered 00000100:00000001:5.0:1496184950.997056:0:17513:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:5.0:1496184950.997059:0:17513:0:(sec.c:2099:sptlrpc_svc_unwrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:5.0:1496184950.997060:0:17513:0:(service.c:1950:ptlrpc_server_handle_req_in()) got req x1568861102817424 00000020:00000001:5.0:1496184950.997062:0:17513:0:(genops.c:722:class_conn2export()) Process entered 00000020:00000040:5.0:1496184950.997063:0:17513:0:(genops.c:734:class_conn2export()) looking for export cookie 0xb9454cf3a18cf8e9 00000020:00000001:5.0:1496184950.997064:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000040:5.0:1496184950.997065:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 8 00000020:00000001:5.0:1496184950.997067:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000020:00000001:5.0:1496184950.997068:0:17513:0:(genops.c:736:class_conn2export()) Process leaving (rc=18446612201958517760 : -131871751033856 : ffff8810371fac00) 00000100:00000001:5.0:1496184950.997070:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.997071:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000020:00000010:5.0:1496184950.997073:0:17513:0:(lu_object.c:1723:keys_init()) kmalloced 'ctx->lc_value': 320 at ffff881010fb4400. 00000020:00000010:5.0:1496184950.997075:0:17513:0:(lu_ucred.c:49:lu_ucred_key_init()) kmalloced 'value': 104 at ffff88100b6eb000. 00000020:00000010:5.0:1496184950.997076:0:17513:0:(tgt_main.c:382:tgt_ses_key_init()) kmalloced 'value': 160 at ffff88104ffc6000. 00000100:00000040:5.0:1496184950.997078:0:17513:0:(service.c:1195:ptlrpc_at_set_timer()) armed ost_io at +38s 00000100:00000001:5.0:1496184950.997080:0:17513:0:(service.c:1667:ptlrpc_server_request_add()) Process entered 00000100:00000001:5.0:1496184950.997080:0:17513:0:(service.c:1591:ptlrpc_server_hpreq_init()) Process entered 00000020:00000001:5.0:1496184950.997081:0:17513:0:(tgt_handler.c:798:tgt_hpreq_handler()) Process entered 00000020:00000001:5.0:1496184950.997082:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.997083:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.997085:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.997086:0:17513:0:(tgt_handler.c:248:tgt_ost_body_unpack()) Process entered 00000020:00000001:5.0:1496184950.997087:0:17513:0:(tgt_handler.c:140:tgt_validate_obdo()) Process entered 00000020:00000001:5.0:1496184950.997088:0:17513:0:(tgt_handler.c:178:tgt_validate_obdo()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.997089:0:17513:0:(nodemap_handler.c:578:nodemap_get_from_exp()) Process entered 00000001:00000001:5.0:1496184950.997090:0:17513:0:(nodemap_handler.c:581:nodemap_get_from_exp()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.997091:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.997092:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.997093:0:17513:0:(nodemap_handler.c:633:nodemap_map_id()) Process entered 00000001:00000001:5.0:1496184950.997094:0:17513:0:(nodemap_handler.c:680:nodemap_map_id()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997095:0:17513:0:(tgt_handler.c:195:tgt_io_data_unpack()) Process entered 00000020:00000001:5.0:1496184950.997096:0:17513:0:(tgt_handler.c:238:tgt_io_data_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997097:0:17513:0:(tgt_handler.c:297:tgt_ost_body_unpack()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997098:0:17513:0:(tgt_handler.c:366:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.997099:0:17513:0:(ofd_dev.c:2711:ofd_hp_brw()) Process entered 00000020:00000001:5.0:1496184950.997100:0:17513:0:(tgt_handler.c:821:tgt_hpreq_handler()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.997101:0:17513:0:(ofd_dev.c:2513:ofd_rw_hpreq_check()) Process entered 00002000:00100000:5.0:1496184950.997104:0:17513:0:(ofd_dev.c:2544:ofd_rw_hpreq_check()) @@@ lustre-OST0001 ll_ost_io01_005: refresh rw locks: [0x100010000:0x463:0x0] (0->1048575) req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00002000:00000001:5.0:1496184950.997110:0:17513:0:(ofd_dev.c:2398:ofd_prolong_extent_locks()) Process entered 00002000:00100000:5.0:1496184950.997111:0:17513:0:(ofd_dev.c:2407:ofd_prolong_extent_locks()) Prolong locks for req ffff88102600e450 with x1568861102817424 ext(0->1048575) 00010000:00000001:5.0:1496184950.997113:0:17513:0:(ldlm_lock.c:584:__ldlm_handle2lock()) Process entered 00000020:00000001:5.0:1496184950.997114:0:17513:0:(lustre_handles.c:156:class_handle2object()) Process entered 00000020:00000001:5.0:1496184950.997115:0:17513:0:(lustre_handles.c:179:class_handle2object()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.997116:0:17513:0:(ldlm_lock.c:603:__ldlm_handle2lock()) Process leaving (rc=18446612201953419264 : -131871756132352 : ffff881036d1e000) 00010000:00000001:5.0:1496184950.997118:0:17513:0:(ldlm_lock.c:188:ldlm_lock_put()) Process entered 00010000:00000001:5.0:1496184950.997118:0:17513:0:(ldlm_lock.c:222:ldlm_lock_put()) Process leaving 00002000:00000001:5.0:1496184950.997119:0:17513:0:(ofd_dev.c:2422:ofd_prolong_extent_locks()) Process leaving 00002000:00010000:5.0:1496184950.997119:0:17513:0:(ofd_dev.c:2549:ofd_rw_hpreq_check()) lustre-OST0001: refreshed 0 locks timeout for req ffff88102600e450. 00002000:00000001:5.0:1496184950.997121:0:17513:0:(ofd_dev.c:2554:ofd_rw_hpreq_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.997122:0:17513:0:(service.c:1618:ptlrpc_server_hpreq_init()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.997124:0:17513:0:(service.c:1706:ptlrpc_server_request_add()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.997128:0:17513:0:(service.c:2014:ptlrpc_server_handle_req_in()) Process leaving (rc=1 : 1 : 1) 00004000:00000001:5.0:1496184950.997131:0:17513:0:(watchdog.c:433:lc_watchdog_disable()) Process entered 00004000:00000001:5.0:1496184950.997131:0:17513:0:(watchdog.c:441:lc_watchdog_disable()) Process leaving 00004000:00000001:5.0:1496184950.997132:0:17513:0:(watchdog.c:417:lc_watchdog_touch()) Process entered 00004000:00000001:5.0:1496184950.997133:0:17513:0:(watchdog.c:427:lc_watchdog_touch()) Process leaving 00000100:00000001:5.0:1496184950.997134:0:17513:0:(service.c:2039:ptlrpc_server_handle_request()) Process entered 00000100:00000001:5.0:1496184950.997135:0:17513:0:(service.c:1820:ptlrpc_server_request_get()) Process entered 00000100:00100000:5.0:1496184950.997137:0:17513:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.37.248.124@o2ib1, seq: 57719 00000100:00000040:5.0:1496184950.997139:0:17513:0:(service.c:1851:ptlrpc_server_request_get()) RPC GETting export ffff8810371fac00 : new rpc_count 1 00000100:00000001:5.0:1496184950.997140:0:17513:0:(service.c:1853:ptlrpc_server_request_get()) Process leaving (rc=18446612201671287888 : -131872038263728 : ffff88102600e450) 00000100:00000040:5.0:1496184950.997142:0:17513:0:(lustre_net.h:2460:ptlrpc_rqphase_move()) @@@ move req "New" -> "Interpret" req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/0 e 0 to 0 dl 1496184993 ref 1 fl New:/2/ffffffff rc 0/-1 00000100:00000001:5.0:1496184950.997148:0:17513:0:(service.c:1060:ptlrpc_update_export_timer()) Process entered 00000100:00000001:5.0:1496184950.997148:0:17513:0:(service.c:1073:ptlrpc_update_export_timer()) Process leaving 00000100:00100000:5.0:1496184950.997150:0:17513:0:(service.c:2100:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost_io01_005:935a409f-52ff-39aa-2890-65a4411c7c57+8:12090:x1568861102817424:12345-10.37.248.124@o2ib1:4 00000100:00000200:5.0:1496184950.997152:0:17513:0:(service.c:2105:ptlrpc_server_handle_request()) got req 1568861102817424 00000020:00000001:5.0:1496184950.997153:0:17513:0:(tgt_handler.c:672:tgt_request_handle()) Process entered 00000020:00000001:5.0:1496184950.997155:0:17513:0:(tgt_handler.c:617:process_req_last_xid()) Process entered 00000020:00000001:5.0:1496184950.997156:0:17513:0:(tgt_handler.c:657:process_req_last_xid()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997157:0:17513:0:(tgt_handler.c:584:tgt_handler_find_check()) Process entered 00000020:00000001:5.0:1496184950.997158:0:17513:0:(tgt_handler.c:611:tgt_handler_find_check()) Process leaving (rc=18446744072120284160 : -1589267456 : ffffffffa145b400) 00000020:00000001:5.0:1496184950.997159:0:17513:0:(tgt_handler.c:523:tgt_handle_recovery()) Process entered 00000020:00000001:5.0:1496184950.997160:0:17513:0:(tgt_handler.c:573:tgt_handle_recovery()) Process leaving (rc=1 : 1 : 1) 00000020:00000001:5.0:1496184950.997161:0:17513:0:(tgt_handler.c:382:tgt_handle_request0()) Process entered 00000020:00000001:5.0:1496184950.997162:0:17513:0:(tgt_handler.c:311:tgt_request_preprocess()) Process entered 00000020:00000001:5.0:1496184950.997162:0:17513:0:(tgt_handler.c:314:tgt_request_preprocess()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997164:0:17513:0:(tgt_handler.c:2134:tgt_brw_write()) Process entered 00000100:00000001:5.0:1496184950.997165:0:17513:0:(pack_generic.c:334:lustre_pack_reply_v2()) Process entered 02000000:00000001:5.0:1496184950.997166:0:17513:0:(sec.c:2112:sptlrpc_svc_alloc_rs()) Process entered 02000000:00000010:5.0:1496184950.997167:0:17513:0:(sec_null.c:318:null_alloc_rs()) kmalloced 'rs': 704 at ffff88100e688000. 02000000:00000001:5.0:1496184950.997169:0:17513:0:(sec.c:2148:sptlrpc_svc_alloc_rs()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.997170:0:17513:0:(pack_generic.c:368:lustre_pack_reply_v2()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997171:0:17513:0:(tgt_handler.c:1622:tgt_brw_lock()) Process entered 00000020:00000001:5.0:1496184950.997172:0:17513:0:(tgt_handler.c:1631:tgt_brw_lock()) Process leaving (rc=0 : 0 : 0) 00000020:00000020:5.0:1496184950.997174:0:17513:0:(tgt_handler.c:2224:tgt_brw_write()) @@@ clear resent/replay req grant info req@ffff88102600e450 x1568861102817424/t0(0) o4->935a409f-52ff-39aa-2890-65a4411c7c57@10.37.248.124@o2ib1:738/0 lens 608/448 e 0 to 0 dl 1496184993 ref 1 fl Interpret:/2/0 rc 0/0 00000020:00000001:5.0:1496184950.997178:0:17513:0:(obd_class.h:1126:obd_preprw()) Process entered 00002000:00000001:5.0:1496184950.997181:0:17513:0:(ofd_io.c:543:ofd_preprw_write()) Process entered 00002000:00000001:5.0:1496184950.997182:0:17513:0:(ofd_objects.c:109:ofd_object_find()) Process entered 00002000:00000001:5.0:1496184950.997184:0:17513:0:(ofd_objects.c:117:ofd_object_find()) Process leaving (rc=18446612166904486528 : -131906805065088 : ffff88080dbd9a80) 00002000:00000001:5.0:1496184950.997186:0:17513:0:(tgt_grant.c:1181:tgt_grant_prepare_write()) Process entered 00002000:00000001:5.0:1496184950.997187:0:17513:0:(tgt_grant.c:282:tgt_statfs_internal()) Process entered 00002000:00000001:5.0:1496184950.997187:0:17513:0:(tgt_grant.c:356:tgt_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0) 00002000:00000020:5.0:1496184950.997189:0:17513:0:(tgt_grant.c:406:tgt_grant_statfs()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 free: 80206888960 avail: 76094111744 00002000:00000001:5.0:1496184950.997191:0:17513:0:(tgt_grant.c:432:tgt_grant_space_left()) Process entered 00002000:00000020:5.0:1496184950.997192:0:17513:0:(tgt_grant.c:468:tgt_grant_space_left()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 avail 76094111744 left 76084350976 unstable 0 tot_grant 9758656 pending 0 00002000:00000001:5.0:1496184950.997194:0:17513:0:(tgt_grant.c:470:tgt_grant_space_left()) Process leaving (rc=76084350976 : 76084350976 : 11b6fb1000) 00002000:00000001:5.0:1496184950.997195:0:17513:0:(tgt_grant.c:495:tgt_grant_incoming()) Process entered 00002000:00000001:5.0:1496184950.997196:0:17513:0:(tgt_grant.c:502:tgt_grant_incoming()) Process leaving 00002000:00000001:5.0:1496184950.997196:0:17513:0:(tgt_grant.c:696:tgt_grant_check()) Process entered 00002000:00000020:5.0:1496184950.997197:0:17513:0:(tgt_grant.c:823:tgt_grant_check()) lustre-OST0001: cli 935a409f-52ff-39aa-2890-65a4411c7c57/ffff8810371fac00 granted: 0 ungranted: 1073152 grant: 9510912 dirty: 0 00002000:00000001:5.0:1496184950.997199:0:17513:0:(tgt_grant.c:828:tgt_grant_check()) Process leaving 00002000:00000001:5.0:1496184950.997200:0:17513:0:(tgt_grant.c:1236:tgt_grant_prepare_write()) Process leaving 00000001:00000001:5.0:1496184950.997201:0:17513:0:(osd_io.c:395:osd_map_remote_to_local()) Process entered 00000001:00000001:5.0:1496184950.997203:0:17513:0:(osd_io.c:421:osd_map_remote_to_local()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:5.0:1496184950.997226:0:17513:0:(osd_io.c:535:osd_bufs_get()) Process leaving (rc=256 : 256 : 100) 00000001:00000001:5.0:1496184950.997232:0:17513:0:(osd_io.c:1049:osd_write_prep()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997233:0:17513:0:(lprocfs_jobstats.c:272:lprocfs_job_stats_log()) Process entered 00000020:00000001:5.0:1496184950.997236:0:17513:0:(lprocfs_jobstats.c:323:lprocfs_job_stats_log()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.997237:0:17513:0:(ofd_io.c:656:ofd_preprw_write()) Process leaving (rc=0 : 0 : 0) 00002000:00000001:5.0:1496184950.997238:0:17513:0:(ofd_io.c:758:ofd_preprw()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:5.0:1496184950.997239:0:17513:0:(obd_class.h:1132:obd_preprw()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:5.0:1496184950.997240:0:17513:0:(niobuf.c:130:ptlrpc_prep_bulk_exp()) Process entered 00000100:00000010:5.0:1496184950.997242:0:17513:0:(client.c:129:ptlrpc_new_bulk()) kmalloced 'desc': 280 at ffff881010fb6e00. 00000100:00000010:5.0:1496184950.997243:0:17513:0:(client.c:134:ptlrpc_new_bulk()) kmalloced '((desc)->bd_u.bd_kiov.bd_vec)': 4096 at ffff881037771000. 00000020:00000040:5.0:1496184950.997245:0:17513:0:(genops.c:818:class_export_get()) GETting export ffff8810371fac00 : new refcount 9 00010000:00000001:5.0:1496184950.997249:0:17513:0:(ldlm_lib.c:3167:target_bulk_io()) Process entered 00000100:00000001:5.0:1496184950.997250:0:17513:0:(niobuf.c:164:ptlrpc_start_bulk_transfer()) Process entered 00000100:00000200:5.0:1496184950.997277:0:17513:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 256 pages 1048576 bytes via portal 8 id 12345-10.37.248.124@o2ib1 mbits 0x592def0cff1c0-0x592def0cff1c0 00000100:00000001:5.0:1496184950.997279:0:17513:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1496184950.998622:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184950.998624:0:6129:0:(events.c:449:server_bulk_callback()) event type 5, status -103, desc ffff881010fb6e00 00000100:00000001:3.0:1496184951.009716:0:6129:0:(events.c:476:server_bulk_callback()) Process leaving 00000100:00000001:3.0:1496184951.009721:0:6129:0:(events.c:438:server_bulk_callback()) Process entered 00000100:00020000:3.0:1496184951.009722:0:6129:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff881010fb6e00 Debug log: 54376 lines, 54376 kept, 0 dropped, 0 bad.